Facing Trouble with perfectly running Prod

These are cross bar Logs on my Local system. Crossbar works properly on local, dev environment (vps), staging environment(vps) but when running on production machine, it doesn’t go past ‘component-001’ as seen in logs below.

Local Logs

2019-08-07T12:46:20+0500 [Controller 19022] Received SIGTERM, shutting down.
2019-08-07T12:53:05+0500 [Controller 5015] Node configuration loaded from 'config.json'
2019-08-07T12:53:05+0500 [Controller 5015] __ __ __ __ __ __ __ __
2019-08-07T12:53:05+0500 [Controller 5015] /  `|__)/ \/__` / **`|** ) /\ | <strong>) |/
2019-08-07T12:53:05+0500 [Controller 5015] __,| \</strong> /. **/.** /|__)/~~| . |__/
2019-08-07T12:53:05+0500 [Controller 5015]
2019-08-07T12:53:05+0500 [Controller 5015] Crossbar.io Version: 16.10.1
2019-08-07T12:53:05+0500 [Controller 5015] Node Public Key: 57fdffc3cdfa9eb98295eb9b7a73285fe9360aa42e7199d71e53118ab1642af9
2019-08-07T12:53:05+0500 [Controller 5015]
2019-08-07T12:53:05+0500 [Controller 5015] Running from node directory '/home/nextgeni/PycharmProjects/HHbackend_LocalDev/backend/ws_app/.crossbar'
2019-08-07T12:53:05+0500 [Controller 5015] Controller process starting (CPython-EPollReactor) ..
2019-08-07T12:53:05+0500 [Controller 5015] Using default node shutdown triggers ['shutdown_on_worker_exit']
2019-08-07T12:53:05+0500 [Controller 5015] Joined realm 'crossbar' on node management router
2019-08-07T12:53:05+0500 [Controller 5015] Configuring node from local configuration ...
2019-08-07T12:53:05+0500 [Controller 5015] Starting 1 workers ...
2019-08-07T12:53:05+0500 [Controller 5015] Starting Router with ID 'worker-001'...
2019-08-07T12:53:06+0500 [Router 5129] Worker process starting (CPython-EPollReactor) ..
2019-08-07T12:53:07+0500 [Controller 5015] Router with ID 'worker-001' and PID 5129 started
2019-08-07T12:53:07+0500 [Router 5129] Realm 'webrtcapp' started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': realm 'realm-001' (named 'webrtcapp') started
2019-08-07T12:53:07+0500 [Router 5129] role role-001 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-001' (named 'anonymous') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-002 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-002' (named 'doctor') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-003 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-003' (named 'patient') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-004 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-004' (named 'backend') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-005 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-005' (named 'authenticator') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-006 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-006' (named 'api_bridge') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-007 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-007' (named 'authorizer') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-008 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-008' (named 'unicator') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] role role-009 on realm realm-001 started
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': role 'role-009' (named 'watcher') started on realm 'realm-001'
2019-08-07T12:53:07+0500 [Router 5129] started component: authenticator.AuthenticatorSession id=3206775862337037
2019-08-07T12:53:07+0500 [Router 5129] Dynamic authenticator registered
2019-08-07T12:53:07+0500 [Controller 5015] Router 'worker-001': component 'component-001' started
2019-08-07T12:53:08+0500 [Router 5129] started component: main.AppSession id=790276721903932
2019-08-07T12:53:18+0500 [Controller 5015] Router 'worker-001': component 'component-002' started
2019-08-07T12:53:18+0500 [Router 5129] started component: authorize.Authorizer id=2684889310063826
2019-08-07T12:53:18+0500 [Router 5129] Dynamic authorizer registered
2019-08-07T12:53:18+0500 [Controller 5015] Router 'worker-001': component 'component-003' started
2019-08-07T12:53:18+0500 [Router 5129] started component: session_unicator.Unicator id=3504627812550999
2019-08-07T12:53:18+0500 [Controller 5015] Router 'worker-001': component 'component-004' started
2019-08-07T12:53:18+0500 [Router 5129] Site starting on 8080
2019-08-07T12:53:18+0500 [Controller 5015] Router 'worker-001': transport 'transport-001' started
2019-08-07T12:53:18+0500 [Router 5129] Loading server TLS key from /home/nextgeni/PycharmProjects/HHbackend_LocalDev/backend/ws_app/.crossbar/ssl/privkey.pem
2019-08-07T12:53:18+0500 [Router 5129] Loading server TLS certificate from /home/nextgeni/PycharmProjects/HHbackend_LocalDev/backend/ws_app/.crossbar/ssl/cert.pem
2019-08-07T12:53:18+0500 [Router 5129] Loading server TLS chain certificate from /home/nextgeni/PycharmProjects/HHbackend_LocalDev/backend/ws_app/.crossbar/ssl/chain.pem

Production Logs

2020-09-22T11:19:48+0000 [Controller 5838] Node configuration loaded from 'config.json'
2020-09-22T11:19:48+0000 [Controller 5838] __ __ __ __ __ __ __ __
2020-09-22T11:19:48+0000 [Controller 5838] /  `|__)/ \/__` / **`|** ) /\ | <strong>) |/
2020-09-22T11:19:48+0000 [Controller 5838] __,| \</strong> /. **/.** /|__)/~~| . |__/
2020-09-22T11:19:48+0000 [Controller 5838]
2020-09-22T11:19:48+0000 [Controller 5838] Crossbar.io Version: 16.10.1
2020-09-22T11:19:48+0000 [Controller 5838] Node Public Key: 57fdffc3cdfa9eb98295eb9b7a73285fe9360aa42e7199d71e53118ab1642af9
2020-09-22T11:19:48+0000 [Controller 5838]
2020-09-22T11:19:48+0000 [Controller 5838] Running from node directory '/home/ubuntu/hah/backend/ws_app/.crossbar'
2020-09-22T11:19:48+0000 [Controller 5838] Controller process starting (CPython-EPollReactor) ..
2020-09-22T11:19:48+0000 [Controller 5838] Using default node shutdown triggers ['shutdown_on_worker_exit']
2020-09-22T11:19:48+0000 [Controller 5838] Joined realm 'crossbar' on node management router
2020-09-22T11:19:48+0000 [Controller 5838] Configuring node from local configuration ...
2020-09-22T11:19:48+0000 [Controller 5838] Starting 1 workers ...
2020-09-22T11:19:48+0000 [Controller 5838] Starting Router with ID 'worker-001'...
2020-09-22T11:19:48+0000 [Router 5860] Worker process starting (CPython-EPollReactor) ..
2020-09-22T11:19:49+0000 [Controller 5838] Router with ID 'worker-001' and PID 5860 started
2020-09-22T11:19:49+0000 [Router 5860] Realm 'webrtcapp' started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': realm 'realm-001' (named 'webrtcapp') started
2020-09-22T11:19:49+0000 [Router 5860] role role-001 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-001' (named 'anonymous') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-002 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-002' (named 'doctor') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-003 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-003' (named 'patient') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-004 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-004' (named 'backend') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-005 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-005' (named 'authenticator') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-006 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-006' (named 'api_bridge') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-007 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-007' (named 'authorizer') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-008 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-008' (named 'unicator') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] role role-009 on realm realm-001 started
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': role 'role-009' (named 'watcher') started on realm 'realm-001'
2020-09-22T11:19:49+0000 [Router 5860] started component: authenticator.AuthenticatorSession id=6830729520736477
2020-09-22T11:19:49+0000 [Router 5860] Dynamic authenticator registered
2020-09-22T11:19:49+0000 [Controller 5838] Router 'worker-001': component 'component-001' started
2020-09-22T11:19:49+0000 [Router 5860] started component: main.AppSession id=785801880505640

Upon detailed inspection, we learnt that this happens whether we use Supervisor or not. One more thing we noticed that when closing with supervisor Crossbar Worker [router] doesn’t close properly and leaves process behind whereas crossbar controller closes down perfectly.

Can you please share your production config (with sensitive stuff redacted), it may just be an issue with the config.

Another thing that may help is if you provide more verbose crossbar logs. Kindly start crossbar with --loglevel debug and paste the logs again.

(My hunch is there might be a blocking call in your authenticator).

already answered and resolved here https://github.com/crossbario/crossbar/issues/1814

tldr: invalid, there is no issue