Information Security & US & Life & Love & Fashion & Hacking & Passion

#!/usr/bin/perl

#perl-grep3.pl

my $pattern = shift @ARGV;
my $regex=eval {qr/$pattern/};
die "Check your pattern! $@" if $@;

while( <> )
        {
        print "$_\t\tmatched >>>$&<<<\n" if m/$regex/;
        }

[root@blackhole:/tmp]$ perldoc -t perl | perl-grep3.pl "\b(\S)\S\1\b"
    support online at <http://www.perl.org/>.
                matched >>>www<<<
        perluniprops        Index of Unicode Version 5.2.0 properties in Perl
                matched >>>.2.<<<
        perl595delta        Perl changes in version 5.9.5
                matched >>>.9.<<<
        perl594delta        Perl changes in version 5.9.4
                matched >>>.9.<<<
        perl593delta        Perl changes in version 5.9.3
                matched >>>.9.<<<
        perl592delta        Perl changes in version 5.9.2
                matched >>>.9.<<<
        perl591delta        Perl changes in version 5.9.1
                matched >>>.9.<<<
        perl590delta        Perl changes in version 5.9.0
                matched >>>.9.<<<
        perl589delta        Perl changes in version 5.8.9
                matched >>>.8.<<<
        perl588delta        Perl changes in version 5.8.8
                matched >>>.8.<<<
        perl587delta        Perl changes in version 5.8.7
                matched >>>.8.<<<
        perl586delta        Perl changes in version 5.8.6
                matched >>>.8.<<<
        perl585delta        Perl changes in version 5.8.5
                matched >>>.8.<<<
        perl584delta        Perl changes in version 5.8.4
                matched >>>.8.<<<
        perl583delta        Perl changes in version 5.8.3
                matched >>>.8.<<<
        perl582delta        Perl changes in version 5.8.2
                matched >>>.8.<<<
        perl581delta        Perl changes in version 5.8.1
                matched >>>.8.<<<
        perl58delta         Perl changes in version 5.8.0
                matched >>>.8.<<<
        perl573delta        Perl changes in version 5.7.3
                matched >>>.7.<<<
        perl572delta        Perl changes in version 5.7.2
                matched >>>.7.<<<
        perl571delta        Perl changes in version 5.7.1
                matched >>>.7.<<<
        perl570delta        Perl changes in version 5.7.0
                matched >>>.7.<<<
        perl561delta        Perl changes in version 5.6.1
                matched >>>.6.<<<
     http://www.perl.org/       the Perl homepage
                matched >>>www<<<
     http://www.perl.com/       Perl articles (O'Reilly)
                matched >>>www<<<
     http://www.cpan.org/       the Comprehensive Perl Archive
                matched >>>www<<<
     http://www.pm.org/         the Perl Mongers
                matched >>>www<<<

#!/usr/bin/perl
#perl-grep2.pl

my $pattern = shift @ARGV;

my $regex = eval { qr/$pattern/ };
die "Check your pattern! $@" if $@;

while (<>)
{
        print if m/$regex/;
}

let's find the repeated pattern from PERLDOC.
[root@blackhole:/tmp]$ perldoc -t perl | perl-grep2.pl "\b(\S)\S\1\b"
    support online at <http://www.perl.org/>.
        perluniprops        Index of Unicode Version 5.2.0 properties in Perl
        perl595delta        Perl changes in version 5.9.5
        perl594delta        Perl changes in version 5.9.4
        perl593delta        Perl changes in version 5.9.3
        perl592delta        Perl changes in version 5.9.2
        perl591delta        Perl changes in version 5.9.1
        perl590delta        Perl changes in version 5.9.0
        perl589delta        Perl changes in version 5.8.9
        perl588delta        Perl changes in version 5.8.8
        perl587delta        Perl changes in version 5.8.7
        perl586delta        Perl changes in version 5.8.6
        perl585delta        Perl changes in version 5.8.5
        perl584delta        Perl changes in version 5.8.4
        perl583delta        Perl changes in version 5.8.3
        perl582delta        Perl changes in version 5.8.2
        perl581delta        Perl changes in version 5.8.1
        perl58delta         Perl changes in version 5.8.0
        perl573delta        Perl changes in version 5.7.3
        perl572delta        Perl changes in version 5.7.2
        perl571delta        Perl changes in version 5.7.1
        perl570delta        Perl changes in version 5.7.0
        perl561delta        Perl changes in version 5.6.1
     http://www.perl.org/       the Perl homepage
     http://www.perl.com/       Perl articles (O'Reilly)
     http://www.cpan.org/       the Comprehensive Perl Archive
     http://www.pm.org/         the Perl Mongers

#!/usr/bin/perl
#perl-grep.pl

my $regex = shift @ARGV;

print "Regex is [$regex]\n";

while ( <> )
        {
        print if m/$regex/;
        }

[root@"+":/tmp]$ perl perl-grep.pl new *.pl
Regex is [new]
my $new_name = "$dirname/$basename";
rename ($old_name, $new_name)
 or warn "Can't rename '$old_name' to '$new_name': $!";
[root@"+":/tmp]$

How to use Linux awk programming and regular expression to read a big log file?

Use the Linux tail command to analysis the log file content, in order to understand log entries pattern.

Using the db2diag.log as an example, each event / incident is initiated with a line that contains date and time:

2008-01-02-10.52.47.720435+480 I1840G300          LEVEL: Event
Then, I use the awk and its regular expression to filter out all log entries that match the particular day and hour of interest:

First, find out the record number of first log entry that match the date and time pattern using its regular expression (RegEx) function:

awk '{if ($1 ~ /2008-01-16-17/){print NR}}' < db2diag.log | head -1
Next, find out the record number of last log entry that match the date and time pattern:

awk '{if ($1 ~ /2008-01-16-17/){print NR}}' < db2diag.log | tail -1
Finally, use awk again to extract or filter all log entries within the range of first and last record numbers that we’ve known from last two steps:

awk '{if (NR >= 7529 && NR <= 8382){print $0}}' < db2diag.log
Because the nature of db2diag.log, the last record number I get from awk doesn’t include the detail of DB2 event / incident happened on that particular time. Thus, I purposely top up the “last record number” (suppose the last record number reported by awk command is 8382, I rest it to be 8390):

awk '{if (NR >= 7529 && NR <= 8390){print $0}}' < db2diag.log >tempfile
If you would like to output the extracted log entries to another temporarily file, just redirect the standard output of awk command to a temp file as you wish (e.g. append >tempfile to the end of last awk command sample).

Brief note about the awk programming syntax used in the sample codes at above:

$1 ~ /2008-01-16-17/ means to check if 1st field/column text pattern matches with the regular expression (i.e. 2008-01-16-17).

Unless the field separator (FS) is specified, awk regards space as field separator by default.

The first field (a.k.a column) of a line (awk treats each line as a record) is denoted as $1, 2nd field as $2, and so forth. The $0 is simply means all the fields/columns of the line/record.

Thus, the combination of awk programming and organized text files can form a simple database system!

The awk regular expression pattern is enclosed by a pair of slash character (/).

The awk RegEx operator for match comparison is a tilde/swung dash character (~). (refer to GNU awk notes on Regular Expression).

print NR is meant to print the record number (NR), i.e. the line number in the log file. To print the number of field/column in a line/record, use NF