Feeds:
Posts
Comments

Archive for the ‘Web Tools’ Category

What is Awstats:-

AWStats is an open source Web analytics reporting tool, suitable for analyzing data from Internet services such as web, streaming media, mail and FTP servers. AWStats parses and analyzes server log files, producing HTML reports. Data is visually presented within reports by tables and bar graphs. Static reports can be created through a command line interface, and on-demand reporting is supported through a web browser CGI program.

AWStats supports most major web server log file formats including Apache,WebStar, IIS and many other common web server log formats.

Compatibility

As Awstats is in perl, you can install Awstats on almost any operating system. It is a very popular server administration tool, with packages available for most Linux distributions. AWStats can be installed on a workstation, such as MS Windows, for local use in situations where log files can be downloaded from a remote server.

Licensing

AWStats is licensed under the GNU General Public License (GPL).

Support

Proper web log analysis tool configuration and report interpretation requires a bit of technical and business knowledge. AWStats support resources include documentation, and user community forums

Security considerations

The on-demand CGI program has been the object of security exploits, as is the case of many CGI programs. Organizations wishing to provide public access to their Web analytics reports should consider generating static HTML reports. The on-demand facility can still be used by restricting its use to internal users. Precautions should be taken against referrer spam (Referrer spam filtering functionality was added in version 6.5).

For more information please refer below link:-

http://awstats.sourceforge.net/

Read Full Post »

Prerequisite

As AWStats is Perl based, a Perl Interpreter need to be installed first to take advantage of AWStats. For instruction on how to install Perl in Windows machine, check out this article.

Download AWStats

Download latest stable release of AWStats from http://www.awstats.org/. Choose the setup package that ended with .exe extension.

Install AWStats

Launch the AWStats setup file by double clicking on the executable that just has been downloaded.

Click “Next” to continue.

Click “I Agree” to continue.

You may want to change the installation folder here. Then click on “Install” to start the installation.

Setup and Configuration for Apache Web Server

AWStats installer will automatically launches the awstats_configure.pl script to do several setup actions. If you’re not using the Windows installer, you can find the script in the AWStats tools directory. To execute it use (if location of perl.exe is within your path environment):

perl awstats_configure.pl

This is what the script does/asks (you can do all these steps manually instead of running awstats_configure.pl (automatically launch by Windows installer) if you prefer):

A) awstats_configure.pl tries to determine your current log format from your Apache web server configuration file httpd.conf (it asks for the path if not found). If you use a common log, awstats_configure.pl will suggest changing it to the NCSA combined/XLF/ELF format (you can use your own custom log format but this predefined log format is often the best choice and makes setup easier).

If you answer yes, awstats_configure.pl will modify your httpd.conf, changing the following directive from:

CustomLog /yourlogpath/yourlogfile common

to

CustomLog /yourlogpath/yourlogfile combined

See the Apache manual for more information on this directive (possibly installed on your server as http://www.mysite.com/manual).

B) awstats_configure.pl will then add, if not already present, the following directives to your Apache configuration file (note that the “/usr/local/awstats/wwwroot” path might differ according to your distribution or OS:

Directives to add to your Apache conf file to allow use of AWStats as a CGI.
Note that path “/usr/local/awstats/” must reflect your AWStats Installation path.

Alias /awstatsclasses “/usr/local/awstats/wwwroot/classes/”
Alias /awstatscss “/usr/local/awstats/wwwroot/css/”
Alias /awstatsicons “/usr/local/awstats/wwwroot/icon/”
ScriptAlias /awstats/ “/usr/local/awstats/wwwroot/cgi-bin/”

This is to permit URL access to scripts/files in AWStats directory.

Options None
AllowOverride None
Order allow,deny
Allow from all

C) if changes were made as indicated in parts A and B, awstats_configure.pl restarts Apache to apply the changes. To be sure the log format change is effective, go to your homepage. This is an example of the type of records you should see inserted in your new log file after Apache was restarted:

62.161.78.75 – – [dd/mmm/yyyy:hh:mm:ss +0000] “GET / HTTP/1.1″ 200 1234 “http://www.from.com/from.html” “Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)”

D) awstats_configure.pl will ask you for a name for the configuration profile file. Enter an appropriate name such as that of your web server or the virtual domain to be analyzed, i.e. mysite.

awstats_configure.pl will create a new file called awstats.mysite.conf by copying the template file awstats.model.conf. The new file location is:
– For Linux/BSD/Unix users: /etc/awstats.
– For Mac OS X, Windows and other operating systems: the same directory as awstats.pl (cgi-bin).

E) awstats_configure.pl ends.

Check through the httpd.conf is configured properly according to steps above. It has been reported that sometime the CustomLog is not changed from common to combined.

Additional Configuration at Main Parameters of awstats.mysite.conf

A) Change Apache log file directory by:

Find : LogFile=”/var/log/apache/access.log”
Replace with : LogFile=”C:/Program Files/Apache Group/Apache2/logs/access_log”
(Important: The setting is for default Apache installation path. If you change your Apache path, please input accordingly. Note that the directory seperator is /)

B) Change log file type (Optional, typically no need to change)
Find : LogType=W
Replace the value with “W” for analyzing web server log files, “S” for a streaming server log file, “M” for mail log files, “F” for ftp log files.

C) Check if LogFormat has the value “1″ (it means “NCSA apache combined/ELF/XLF log format”) or use a personalized log format if you don’t use combined log format.

D) Check if SiteDomain parameter with the main domain name or the intranet web server name used to reach the web site to analyze (Example: http://www.mydomain.com).

E) If you have several possible name for same site, add list in HostAlias parameter:

Find : HostAliases=”localhost 127.0.0.1 REGEX[^.*\.myserver\.com$]”
Replace with : HostAliases=”yoursite.com localhost 127.0.0.1″
(You can include alias names from dynamic DNS accounts such as http://www.mysite.serveftp.net and etc.)

F) AWStats statistics can be updated from a browser, to provide real-time statistics, by clicking the “Update now” link that appears when AWStats is used as a CGI. To enable it:

Find : AllowToUpdateStatsFromBrowser=0
Replace with : AllowToUpdateStatsFromBrowser=1

G) Edit ‘awstats.pl’ located in “C:\Program Files\AWStats\wwwroot\cgi-bin\” and change the path on the first line to point to exact location of perl.exe in your system, i.e. #!C:\perl\bin\perl.exe.

To Maintain Just Single CGI Directory – Optional

New version of AWStats creates alias directory entries at the bottom of Apache httpd.conf file. If you want to have just one CGI executable directory, you have to move the AWStats scripts to existing CGI directory.

The following assume installation in default path.

* Copy ALL files and sub-folders contained within “C:\Program Files\AWStats\wwwroot\cgi-bin\” and paste into your Apache2 web server’s cgi-bin at “C:\Program Files\Apache Group\Apache2\cgi-bin\”
* Copy ALL files and sub-folders contained within “C:\Program Files\AWStats\wwwroot\icon\” and paste into our Apache2 web server’s icons folder at “C:\Program Files\Apache Group\Apache2\icons\”

Edit again the awstats.mysite.conf file, the change the following:

Find : DirIcons=”/icon”
Replace with : DirIcons=”/icons”

Delete from Apache web server’s httpd.conf the alias directory that has been inserted by AWStats setup program.

Optional Setup Section

Find : DefaultFile=”index.html”
Replace with your default index page, only if necessary, i.e. DefaultFile=”index.php” or DefaultFile=”index.php index.html”

Find : SkipHosts=””
Replace with : SkipHosts=”127.0.0.1 REGEX[^192\.168\.] REGEX[^10\.0\.]”
This is to not include statistics from your own local machines, they will remain in the actual log files but excluded from awstats counts. To get actual hits and figures it’s sometime necessary, but is not essential at this stage.

Build/Update Statistics

There are two possible way to analyze, update and generate the AWStats statistics, command line and browser based.

Command Line Option

Go to the command line, and change directory to your AWStats cgi-bin directory, i.e. C:\Program Files\AWStats\wwwroot\cgi-bin.

The AWStats update command line is (replace with your domain):

perl awstats.pl -config=www.yourdomain.com -update

AWStats database files are saved in directory defined by DirData parameter in config file, which we left as default (current directory). Update might take a while, if the log file is big. When update is finished, you should get on screen a result like this:

Update for config “/etc/awstats/awstats.myvirtualhostname.conf”
With data in log file “/pathtoyourlog/yourlog.log”…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 225730
Found 122 dropped records,
Found 87 corrupted records,
Found 0 old records,
Found 225521 new qualifed records.

Browser Option

AWStats statistics can be updated from a browser, to provide real-time statistics, by clicking the “Update now” link that appears when AWStats is used as a CGI (only applicable if AllowToUpdateStatsFromBrowser is set to 1 in AWStats config file ).

Read Statistics

Again, there is 2 methods where AWStats’s statistics can be read, corresponding to the generation method of the stats mentioned above.

Command Line Option

At the same directory, after the first update command is finished, run the following command:

perl awstats.pl -config=myvirtualhostname -output -staticlinks > awstats.myvirtualhostname.html

The process may take a while, after it’s finished, a html will be generated where you can view it with a browser by double clicking on it.

Browser Option

Access the following directory (replace with your own domain):

http://www.myserver.mydomain/awstats/awstats.pl?config=myvirtualhostname

To see the analysis results, you have several options depending on your security policy.

Note: you must have created a statistics data base for the analysis period by processing your log files before you try to create reports. See the previous section.

1. The first option is to build the main reports, in a static HTML page, from the command line, using the following syntax (skip to the second option if you only have CGI access):

perl awstats.pl -config=mysite -output -staticlinks > awstats.mysite.html

where mysite must be substituted with the domain/virtual host name you selected earlier during AWStats configuration.

To create specific individual reports, specify the report name on the command line as follows¹:
perl awstats.pl -config=mysite -output=alldomains -staticlinks > awstats.mysite.alldomains.html
perl awstats.pl -config=mysite -output=allhosts -staticlinks > awstats.mysite.allhosts.html
perl awstats.pl -config=mysite -output=lasthosts -staticlinks > awstats.mysite.lasthosts.html
perl awstats.pl -config=mysite -output=unknownip -staticlinks > awstats.mysite.unknownip.html
perl awstats.pl -config=mysite -output=alllogins -staticlinks > awstats.mysite.alllogins.html
perl awstats.pl -config=mysite -output=lastlogins -staticlinks > awstats.mysite.lastlogins.html
perl awstats.pl -config=mysite -output=allrobots -staticlinks > awstats.mysite.allrobots.html
perl awstats.pl -config=mysite -output=lastrobots -staticlinks > awstats.mysite.lastrobots.html
perl awstats.pl -config=mysite -output=urldetail -staticlinks > awstats.mysite.urldetail.html
perl awstats.pl -config=mysite -output=urlentry -staticlinks > awstats.mysite.urlentry.html
perl awstats.pl -config=mysite -output=urlexit -staticlinks > awstats.mysite.urlexit.html
perl awstats.pl -config=mysite -output=browserdetail -staticlinks > awstats.mysite.browserdetail.html
perl awstats.pl -config=mysite -output=osdetail -staticlinks > awstats.mysite.osdetail.html
perl awstats.pl -config=mysite -output=unknownbrowser -staticlinks > awstats.mysite.unknownbrowser.html
perl awstats.pl -config=mysite -output=unknownos -staticlinks > awstats.mysite.unknownos.html
perl awstats.pl -config=mysite -output=refererse -staticlinks > awstats.mysite.refererse.html
perl awstats.pl -config=mysite -output=refererpages -staticlinks > awstats.mysite.refererpages.html
perl awstats.pl -config=mysite -output=keyphrases -staticlinks > awstats.mysite.keyphrases.html
perl awstats.pl -config=mysite -output=keywords -staticlinks > awstats.mysite.keywords.html
perl awstats.pl -config=mysite -output=errors404 -staticlinks > awstats.mysite.errors404.html

¹If you prefer, you can use the awstats_buildstaticpages tool to build all these pages in one command, or to generate PDF files.

Notes:

a) You can also add a filter on the following reports: urldetail, urlentry, urlexit, allhosts, refererpages. The filter can be a regexp (regular expression) on the full key you want AWStats to report on and is appended to the output parameter separated by a “:”.

For example, to output the urldetail report, including only pages which contain /news in their URL, you can use the following command line:
perl awstats.pl -config=mysite -output=urldetail:/news -staticlinks > awstats.mysite.urldetailwithfilter.html

b) If you want to build a report for a particular month, add the options -month=MM -year=YYYY where MM is the month expressed as two digits, i.e. 03, and year is the four digit year. To build a report for a full year, add the options -month=all -year=YYYY (warning: this is often resource intensive and might use a lot of memory and CPU. Unix/Linux like operating systems might benefit from use of the “nice” command.)

2) The second option is to dynamically view your statistics from a browser. To do this, use the URL:
http://www.myserver.mydomain/awstats/awstats.pl?config=mysite

where mysite specifies the configuration file to use (AWStats will use the file awstats.mysite.conf).

All output command line options (except -staticlinks and -logfile) are also available when using AWStats with a browser. Just use them as URL parameters: change “-option” to “&option”, i.e. http://www.myserver.mydomain/awstats/awstats.pl?month=MM&year=YYYY&output=unknownos

Reports are generated in real time from the statistics data base. If this is slow, or putting too much load on your server, consider generating static reports instead.

If the AllowToUpdateStatsFromBrowser parameter is set to 1 in AWStats configuration file, you will also be able to run the update process from your browser. Just click on the link “Update now”.

Read Full Post »

AWStats Requirements
– Perl 5.0.2 or above
– Server must log files in a log file you can read
– System must be able to run perl scripts

Downloading AWStats

(1) wget http://nchc.dl.sourceforge.net/sourceforge/awstats/awstats-6.7.tar.gz

Installing AWStats
You need root access to install awstats. Unzip the downloaded file.

(2) tar -xvzf awstats-6.7.tar.gz

AWStats recommends installing the software at /usr/local/awstats, although you can install it anywhere you desire.

mv awstats-6.7 /usr/local/
(3) cd /usr/local/
(4) mv awstast-6.7 awstats
(5) cd awstats/tools/
(6) perl awstats_config.pl — This will run the script, feel the details & note down final results.

8. mkdir /var/lib/awstats

9. chmod 655 /var/lib/awstats/

10. Take backup of yout httpd.conf file (If in case anything goes wrong)
11. vi /usr/local/apache2/conf/httpd.conf

Add following lines in httpd.conf

+++++++————————————————++++++++++
# Directives to allow use of AWStats as a CGI
#
Alias /awstatsclasses “/usr/local/awstats/wwwroot/classes/”
Alias /awstatscss “/usr/local/awstats/wwwroot/css/”
Alias /awstatsicons “/usr/local/awstats/wwwroot/icon/”
ScriptAlias /awstats/ “/usr/local/awstats/wwwroot/cgi-bin/”

#
# This is to permit URL access to scripts/files in AWStats directory.
#

Options None
AllowOverride None
Order allow,deny
Allow from all

+++++++————————————————–+++++++++

12. vi /etc/awstats/awstats.your.server.com.conf

Change the log file location as per your configuration

##LogFile=”/usr/local/apache2/logs/access_log

13. /usr/local/apache2/bin/apachectl stop
14 ./usr/local/apache2/bin/apachectl start

15. Now run “/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=” #This will take some time to complete this process.

15. DONE now check “http://domainname/awstats/awstats.pl?config= #what you configured.

Read Full Post »