cronolog & virtual hosting

[andrew@xxxxxxxxxxxxxxxxxx]

Andrew Houghton writes:
 > Andrew --
 > 
 > I'm using cronolog but running out of file descriptors.  Since I know that
 > I'm using the common log format with a virtualhost specifier at the start, I
 > was planning on adding some code to cronolog to parse that out and
 > intelligently pass the log lines into the right file -- something like:
 > 
 > CustomLog "|cronolog /virtualhosts/%V/logs/combined.%Y.%m.log"
 > 
 > I can't believe I'm the first to need this or think of it, so I assume I'm
 > missing something obvious.  Is there a reason that this wouldn't work?

Andrew 

[Your mail suggests possibilities that may be of interest to other
cronolog users so I am CC'ing the mailing list]

Your suggested syntax will not work with with cronolog as its stands.
Cronolog does not examine the contents of the log messages.

However it could be made to do so if it was told where to find the
information.

In fact if the approach is generalized then the information used need
not be the virtual host -- it could for example be the value of an
environment variable.  I might be inclined to use the specifiers %0
through to %9 for fields 0 through 9 -- these would not conflict with
the existing specifiers (whereas %V does for some versions of
strftime).  Further fields could be specified as %{10}, %{11} etc.  An
added nicety would be to allow a modifier to indicate that the field
should be omitted on output (such as "-").  So if you had:

    SetEnv   LOGTYPE=log
    SetEnvIf Request_URI \.gif$  LOGTYPE=imagelog
    SetEnvIf Request_URI \.jpeg$ LOGTYPE=imagelog
    SetEnvIf Request_URI \.png$  LOGTYPE=imagelog

    LogFormat "%V %{TYPE}e %h %l %u %t \"%r\" %>s %b" vhostspecial
    CustomLog "|cronolog /virtualhosts/%-0/logs/combined.%Y.%m.%-1" vhostspecial

This would cause Apache to write log entries through the pipe to
cronolog with each entry containing the virtual host name and log type
as the first two fields.  Cronolog would then extract those two fields
and interpolate the values in the log file name, and also remove the
fields (coallescing the field separators) before writing the entry --
thus in this case only common log format entries would be written, but
to separate log files for each host and for images and non-image
resources.

As all virtual hosts could use the same log entry (specified outside
any VirtualHost container) *** ALL SERVERS COULD THEN USE THE SAME
CRONOLOG PROCESS ***  (which is one of the main problems of using
cronolog with lots of virtual hosts, so excuse the shouting).

There are a couple of implementation details that would need to be
thought through.

Firstly cronolog would have know how to parse fields -- some are
enclosed in quotes and the timestamp is enclosed in square brackets.

Secondly one would probably want to ensure that cronolog does not
write to directories outside the specified directory hierarchy, so it
should not use field values if they contained the directory separator
character -- or perhaps it should replace those characters with an
escape code, e.g. %2F.

Thirdly it is likely that successive log entries would be destined for
different files, so it would be useful to have some form of cacheing
of file descriptors, otherwise cronolog would be forever opening and
closing files.  I have some code already written for an experimental
module version of cronolog -- it may well be possible to adapt this.

I will be unable to look at this for a couple of weeks but if you do
try coding it yourself I would be interested in seeing your solution.

Regards
Andrew
-- 
Andrew Ford,  Director       Ford & Mason Ltd             Tel: +44 1531 829900
A.Ford@xxxxxxxxxxxxxxxx      South Wing, Compton House    Fax: +44 1531 829901
http://ford-mason.co.uk      Compton Green, Redmarley  Mobile: +44 7785 258278
http://pauntley-press.co.uk  Gloucester, GL19 3JB
http://refcards.com          Great Britain