Here’s a little howto on merging multiple apache log files when you are migrating servers or want to consolidate your logs into one place. This works on ubuntu and debian, but your mileage may vary on other OS’s.
Here’s the howto:
- Login to old server
- scp old log files over, to the admin home directory, in /logs
# scp -r /home/username/logs/ username@yournewserver:
- If copying multiple directories, and you want to call them something other than logs do this:
# scp -r /home/username/logs/ username@yournewserver:/home/username/logs_domainname/
- Login to new server
- Go to /home/username/logs (or /home/username/logs_domainname)
- Make a backup of current logs
# mkdir new_logs
# cp access_log error_log ./new_logs
- Make a few directories
# mkdir old_logs
# mkdir concat_logs
- Enter the old logs directory, and copy the old log files over
# cd old_logs
# cp /home/admin/logs/* .
- Now we’re going to concatenate all these to a new single file. Logresolvemerge.pl is part of the awstats distribution and has been copied to /usr/local/bin.
# logresolvemerge.pl ./access_log* > access_log_overall_old
# logresolvemerge.pl ./error_log* > error_log_overall_old
- Now we’ll leave this directory, and enter the concat_logs directory
# cd ..
# cd concat_logs
- Now we’ll copy our newly created log files over to this directory
# cp ../old_logs/error_log_overall_old ../old_logs/access_log_overall_old .
- Now we’ll copy in the “new” log files from the new production server. This step can be omitted if there are no stats on this server already, or they are not needed.
# cp ../new_logs/* .
# ls -l
-rw-r–r– 1 root root 554 Sep 2 19:38 access_log
-rw-r–r– 1 root root 3702368 Sep 2 19:37 access_log_overall_old
-rw-r–r– 1 root root 115 Sep 2 19:38 error_log
-rw-r–r– 1 root root 907085 Sep 2 19:37 error_log_overall_old
- Here we’ll combine the old logfile, with the new logfile. This again, can be omitted if not using old stats and new stats.
# logresolvemerge.pl ./access_log ./access_log_overall_old > access_log_final
# logresolvemerge.pl ./error_log ./error_log_overall_old > error_log_final
- Now we’ll either have in our directory, a final old log file, or a final combined file from this server, and the old server up to the minute.
If you didn’t do step 11 and 12, all that’s left is:
# cp ./access_log_overall_old ../access_log
# cp ./error_log_overall_old ../error_log
# /etc/init.d/apache restart
If you did do step’s 11 and 12, then do this:
# cp ./access_log_final ../access_log
# cp ./error_log__final ../error_log
# /etc/init.d/apache restart
- You can now delete the old_logs, new_logs, concat_logs directories if wanted.
# cd /home/domain/logs/
# rm -rf concat_logs
# rm -rf new_logs
# rm -rf old_logs
- To update the logs, and make them visible on the awstats interface run this:
# /usr/lib/cgi-bin/awstats.pl -config=yourdomain.tld -update wq
- Visit them on the web at;
Add the update line to your crontab to schedule it. You can also use updateall.pl that comes with the awstats distribution to update all sites at one time if vhosting.
- Go have a rum and coke to celebrate!