sisinfo module to parse httpd logs

[dchaplinsky]

Hi Ali.

I wrote a small implementation of pysisinfo module which is able to grab some useful stats like avg. time for rendering one request and amount of visitors generated by search bots from nginx/apache logs.

Take a look: https://gist.github.com/1927921

Can be helpful to determine what caused a higher load on your server or profile effect of recent changes in codebase of website on average generation time

It's not really an issue but I'm not able to send you personal message via github

[aouyar]

Hi Dimitry,

I just had time to have a closer look at the code you have posted.

Plugins that parse logs could be great, but I was think that the logtail (http://www.linuxcertif.com/man/8/logtail/) approach for processing the records can be more efficient, accurate and easier to use. Logtail avoids parsing the records that has already been parsed by recording the offset of the last record read in each run. Your approach for generating statistics for the log records from last 5 minutes can also work, but implies tuning the parameter tail_records manually to a value that would avoid parsing too many records and and the same time guarantee that all records from the last 5 minutes can be parsed.

I've just checked the Perl code for logtail and it is actually very simple, and I would probably reimplement it in Python so that the log parsing code would not be dependent on any external executable; neither tail nor logtail. The log parsing module can record the log offset and the accumulated stats in an external file at each run to avoid having to reparse records. Once we have a generic log parsing module, more specific modules for parsing Apache, Nginx, Postfix, etc. could be developed easily.

The reason you are not able to send me a personal message via GitHub is probably because you have not provided an e-mail notification address in GitHub.

Thanks for your contributions, Ali