Crossbar 10.4 Python Logging

#1

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

	"pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes

#2

Update - on my MAC OS with Crossbar 10.1 logging works just fine. On Ubuntu with crossbar 10.4 files are created but nothing is written to them.

···

On Wednesday, July 15, 2015 at 4:56:45 PM UTC-7, Jim Benedetto wrote:

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

  "pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes

#3

Hi Jim!

There are some fixes in the upcoming release which hopefully fix this. Shouldn’t be long before we release this. For now: Sorry for the inconvenience.

Regards,

Alex

···

Am Donnerstag, 16. Juli 2015 01:59:38 UTC+2 schrieb Jim Benedetto:

Update - on my MAC OS with Crossbar 10.1 logging works just fine. On Ubuntu with crossbar 10.4 files are created but nothing is written to them.

On Wednesday, July 15, 2015 at 4:56:45 PM UTC-7, Jim Benedetto wrote:

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

  "pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes

#4

Hi Alex - thank you for the fast response - I would have spent another 10 hours looking online bc I thought I was doing something wrong. Two questions for you:

  1. I am on 10.4 right now - is there a later version I should be using?

  2. Once the fix is released what is considered the ‘best practices’ way to log from Crossbar.io when using Python Twisted? I have quite a few components, all of type:Class / all of them written in native python and ApplicationSession objects. What I would like to do would be -

  • Have each component have its own log file where application specific messages can be logged / no Wamp or Crossbar specific messages in these files

  • Have a single “main” log file that either only has Wamp and Crossbar messages, or has all the logs (Wamp/Crossbar/and Application - basically what you see in standard out when you just run crossbar.

  1. Do you know why I am not encountering this issue in my Mac development environment, but am in my production Ubuntu environment? The only difference I have found is on my Mac I am using python 2.7.10 and on Ubuntu I am using python 2.7.6. Upgrading ubuntu to 2.7.10 isn’t easy but if it would fix the issue I would be happy to do it. I’m trying to figure out why the logging is working fine on the Mac in order to determine if there are any workarounds I can use until the fix is released.

Thanks again for your quick response. I have been using Autobahn since the early days of WAMP1 in large scale production Internet of Things systems with huge numbers of sensors all communicating with each other and it is an absolutely rock solid product. The performance and stability have been outstanding.

Jim

···

On Thursday, July 16, 2015 at 6:29:27 AM UTC-7, Alexander Gödde wrote:

Hi Jim!

There are some fixes in the upcoming release which hopefully fix this. Shouldn’t be long before we release this. For now: Sorry for the inconvenience.

Regards,

Alex

Am Donnerstag, 16. Juli 2015 01:59:38 UTC+2 schrieb Jim Benedetto:

Update - on my MAC OS with Crossbar 10.1 logging works just fine. On Ubuntu with crossbar 10.4 files are created but nothing is written to them.

On Wednesday, July 15, 2015 at 4:56:45 PM UTC-7, Jim Benedetto wrote:

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

  "pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes

#5

HI Jim!

  1. Crossbar.io is moving fast, and the last PyPi release is a while back. There has been a lot of work on logging in the meantime. You can install from the crossbar github repo (clone and do make install) to get the latest version.

Regarding your other logging-related questions, our developer hawkowl is the expert here. I’m forwarding your questions to her.

Regards,

Alex

···

Am Donnerstag, 16. Juli 2015 21:11:17 UTC+2 schrieb Jim Benedetto:

Hi Alex - thank you for the fast response - I would have spent another 10 hours looking online bc I thought I was doing something wrong. Two questions for you:

  1. I am on 10.4 right now - is there a later version I should be using?
  1. Once the fix is released what is considered the ‘best practices’ way to log from Crossbar.io when using Python Twisted? I have quite a few components, all of type:Class / all of them written in native python and ApplicationSession objects. What I would like to do would be -
  • Have each component have its own log file where application specific messages can be logged / no Wamp or Crossbar specific messages in these files
  • Have a single “main” log file that either only has Wamp and Crossbar messages, or has all the logs (Wamp/Crossbar/and Application - basically what you see in standard out when you just run crossbar.
  1. Do you know why I am not encountering this issue in my Mac development environment, but am in my production Ubuntu environment? The only difference I have found is on my Mac I am using python 2.7.10 and on Ubuntu I am using python 2.7.6. Upgrading ubuntu to 2.7.10 isn’t easy but if it would fix the issue I would be happy to do it. I’m trying to figure out why the logging is working fine on the Mac in order to determine if there are any workarounds I can use until the fix is released.

Thanks again for your quick response. I have been using Autobahn since the early days of WAMP1 in large scale production Internet of Things systems with huge numbers of sensors all communicating with each other and it is an absolutely rock solid product. The performance and stability have been outstanding.

Jim

On Thursday, July 16, 2015 at 6:29:27 AM UTC-7, Alexander Gödde wrote:

Hi Jim!

There are some fixes in the upcoming release which hopefully fix this. Shouldn’t be long before we release this. For now: Sorry for the inconvenience.

Regards,

Alex

Am Donnerstag, 16. Juli 2015 01:59:38 UTC+2 schrieb Jim Benedetto:

Update - on my MAC OS with Crossbar 10.1 logging works just fine. On Ubuntu with crossbar 10.4 files are created but nothing is written to them.

On Wednesday, July 15, 2015 at 4:56:45 PM UTC-7, Jim Benedetto wrote:

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

  "pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes

#6

Hi Jim!

  1. The in-development version, 0.11, is available from GitHub (https://github.com/crossbario/crossbar).
  2. The best practice is to use the new Twisted Logger http://twistedmatrix.com/documents/current/core/howto/logger.html . If you instantiate one, it will automatically publish events to the global logger that Crossbar uses. If you want to have just your log entries in a separate file, you will have to do something like the following:

from twisted.logger import LogPublisher, Logger, textFileLogObserver, globalLogPublisher

myLogFile = textFileLogObserver(open("/path/to/whereyouwant.log", “w”))
myAppPublisher = LogPublisher(globalLogPublisher, myLogFile)

myLogger = Logger(observer=myAppPublisher)
myLogger.info(“This message will go to both Crossbar’s global log and also your configured file.”)

This setup here writes logs to both the global log (so it will show up in Crossbar’s log), but also get saved to a file. There is also jsonFileLogObserver, which writes out the logs in a more structured format for later analysis (the new Twisted Logger is a ‘structured’ logger, you can pass it complex objects). We’re planning further work on per-component logging, so there might be an easier way to do this in future.

Also, If you want to save the global Crossbar log to a file, crossbar start --help once you’ve installed 0.11 will show the relevant options.

  1. I unfortunately don’t know why it’s happening here. Please do see if 0.11 fixes the issue, if not, please file an Issue on GitHub (https://github.com/crossbario/crossbar/issues) and I’ll try and get to the bottom of it.

Hope this helps.

  • Amber Brown (HawkOwl)
···

On Friday, 17 July 2015 03:11:17 UTC+8, Jim Benedetto wrote:

Hi Alex - thank you for the fast response - I would have spent another 10 hours looking online bc I thought I was doing something wrong. Two questions for you:

  1. I am on 10.4 right now - is there a later version I should be using?
  1. Once the fix is released what is considered the ‘best practices’ way to log from Crossbar.io when using Python Twisted? I have quite a few components, all of type:Class / all of them written in native python and ApplicationSession objects. What I would like to do would be -
  • Have each component have its own log file where application specific messages can be logged / no Wamp or Crossbar specific messages in these files
  • Have a single “main” log file that either only has Wamp and Crossbar messages, or has all the logs (Wamp/Crossbar/and Application - basically what you see in standard out when you just run crossbar.
  1. Do you know why I am not encountering this issue in my Mac development environment, but am in my production Ubuntu environment? The only difference I have found is on my Mac I am using python 2.7.10 and on Ubuntu I am using python 2.7.6. Upgrading ubuntu to 2.7.10 isn’t easy but if it would fix the issue I would be happy to do it. I’m trying to figure out why the logging is working fine on the Mac in order to determine if there are any workarounds I can use until the fix is released.

Thanks again for your quick response. I have been using Autobahn since the early days of WAMP1 in large scale production Internet of Things systems with huge numbers of sensors all communicating with each other and it is an absolutely rock solid product. The performance and stability have been outstanding.

Jim

On Thursday, July 16, 2015 at 6:29:27 AM UTC-7, Alexander Gödde wrote:

Hi Jim!

There are some fixes in the upcoming release which hopefully fix this. Shouldn’t be long before we release this. For now: Sorry for the inconvenience.

Regards,

Alex

Am Donnerstag, 16. Juli 2015 01:59:38 UTC+2 schrieb Jim Benedetto:

Update - on my MAC OS with Crossbar 10.1 logging works just fine. On Ubuntu with crossbar 10.4 files are created but nothing is written to them.

On Wednesday, July 15, 2015 at 4:56:45 PM UTC-7, Jim Benedetto wrote:

Hi -

I am using crossbar 10.4 python and am having a really hard time logging from different modules to different files. Everything is getting dumped to stdout, but using the standard python logging module creates empty files. I have tried to use twisted logging, and am passing setStdout=False, but I end up getting anything that is printed and not just what I directly log with the twisted log module.

I’ve been searching for 6 hours online and can’t find any documentation on the best way to have multiple log files (different ones for different components). A copy of part of my config.json is below, ive tried pythonbuffered, stdout, stdin (stdout and stdin give me this error btw -

Error: encountered unknown attribute ‘stderr’ in ‘options’ in worker configuration.

What I really want to do is just use the normal python logging library, I can make sure I dont do any blocking logging calls. I’ve even tried twisted observers, checked file and directory permissions. I’m out of ideas.

Can anyone help?

Tx

jim

  "pythonpath": [".."],

“env”: {

“vars”: {“PYTHONUNBUFFERED”: “1”}

},

“stdout”: “log”,

“stderr”: “log”

},

0 Likes