Category: Ubuntu

How to Merge Multiple Apache Log Files when Migrating Servers.

Here’s a little howto on merging multiple apache log files when you are migrating servers or want to consolidate your logs into one place. This works on ubuntu and debian, but your mileage may vary on other OS’s.

Here’s the howto:

  1. Login to old server
  2. scp old log files over, to the admin home directory, in /logs

    # scp -r /home/username/logs/ username@yournewserver:

  3. If copying multiple directories, and you want to call them something other than logs do this:

    # scp -r /home/username/logs/ username@yournewserver:/home/username/logs_domainname/

  4. Login to new server
  5. Go to /home/username/logs (or /home/username/logs_domainname)
  6. Make a backup of current logs

    # mkdir new_logs
    # cp access_log error_log ./new_logs

  7. Make a few directories

    # mkdir old_logs
    # mkdir concat_logs

  8. Enter the old logs directory, and copy the old log files over

    # cd old_logs
    # cp /home/admin/logs/* .

  9. Now we’re going to concatenate all these to a new single file. Logresolvemerge.pl is part of the awstats distribution and has been copied to /usr/local/bin.

    # logresolvemerge.pl ./access_log* > access_log_overall_old
    # logresolvemerge.pl ./error_log* > error_log_overall_old

  10. Now we’ll leave this directory, and enter the concat_logs directory

    # cd ..
    # cd concat_logs

  11. Now we’ll copy our newly created log files over to this directory

    # cp ../old_logs/error_log_overall_old ../old_logs/access_log_overall_old .

  12. Now we’ll copy in the “new” log files from the new production server. This step can be omitted if there are no stats on this server already, or they are not needed.

    # cp ../new_logs/* .
    # ls -l

    -rw-r–r– 1 root root 554 Sep 2 19:38 access_log
    -rw-r–r– 1 root root 3702368 Sep 2 19:37 access_log_overall_old
    -rw-r–r– 1 root root 115 Sep 2 19:38 error_log
    -rw-r–r– 1 root root 907085 Sep 2 19:37 error_log_overall_old

  13. Here we’ll combine the old logfile, with the new logfile. This again, can be omitted if not using old stats and new stats.

    # logresolvemerge.pl ./access_log ./access_log_overall_old > access_log_final
    # logresolvemerge.pl ./error_log ./error_log_overall_old > error_log_final

  14. Now we’ll either have in our directory, a final old log file, or a final combined file from this server, and the old server up to the minute.

    If you didn’t do step 11 and 12, all that’s left is:

    # cp ./access_log_overall_old ../access_log
    # cp ./error_log_overall_old ../error_log
    # /etc/init.d/apache restart

    If you did do step’s 11 and 12, then do this:

    # cp ./access_log_final ../access_log
    # cp ./error_log__final ../error_log
    # /etc/init.d/apache restart

  15. You can now delete the old_logs, new_logs, concat_logs directories if wanted.

    # cd /home/domain/logs/
    # rm -rf concat_logs
    # rm -rf new_logs
    # rm -rf old_logs

  16. To update the logs, and make them visible on the awstats interface run this:

    # /usr/lib/cgi-bin/awstats.pl -config=yourdomain.tld -update wq

  17. Visit them on the web at;

    http://www.domainname.com/awstats/awstats.pl

  18. Add the update line to your crontab to schedule it. You can also use updateall.pl that comes with the awstats distribution to update all sites at one time if vhosting.

  19. Go have a rum and coke to celebrate!

Ubuntu Server LTS 8.04 VMWare VMTools Howto and Tutorial

You may have some difficulty with Ubuntu Server and VMWare Tools. Here’s how we got it working in the end:

  • start vm
  • # sudo mount /cdrom
  • # cd /cdrom
  • # cp *.tgz /tmp
  • # cd /tmp
  • # apt-get install build-essentials
  • # apt-get install linux-headers-`uname -r`
  • # tar xzvf vmtools<yourversion.tgz>
  • # cd vmtools<yourver>
  • # perl vmware-install.pl

That’s it. You might want to modify your console settings so you can get 1024×768 from VMWare Workstation, but it’s not really required if you’re just SSH’ing to it.