Question: Can I change the value of the DD and %d wildcards?
Answer: Recently I spoke to a client who was using a separate machine to process his log files. At various times during the day Urchin would FTP over the log file and process the data. The client wanted to use wildcards in the log source so Urchin would always pick up the current day’s file. The client set up the following log file path:
When the client configured the scheduler to run he noticed that Urchin would never get the current days log file. Instead it would always pull the previous days log file.
The reason why Urchin would not retrieve the current file is caused by the way Urchin interprets the DD wildcard and they typical way that log files are rotated. Urchin, by default, converts the DD wildcard to the previous day. So if today is the 20th, Urchin would interpret DD as the 19th. Usually a web server set up for daily log rotation creates a new log file right around midnight. Urchin is usually configured to process the log file sometime after midnight. In this situation, if Urchin was not configured to turn back the clock 24 hours it would pull the current day’s log file. By automatically changing the current time by negative 24 hours Urchin insures that the previous day’s log file is processed.
So, how did we get around this issue? Urchin provides a mechanism to change the data offset. When looking at the data for a log source, click on the Advanced Settings tab. At the bottom of the page there is a section named Date/Time Wildcard Substitution in Log Path Name. Using the Hours edit box you can specify a plus or minus offset in hours. Because Urchin defaults to -24 hours, we entered a +24 to cause the DD wildcard to resolve to the current day.