How to use Linux awk programming and regular expression to read a big log file?

Use the Linux tail command to analysis the log file content, in order to understand log entries pattern.

Using the db2diag.log as an example, each event / incident is initiated with a line that contains date and time:

2008-01-02-10.52.47.720435+480 I1840G300          LEVEL: Event
Then, I use the awk and its regular expression to filter out all log entries that match the particular day and hour of interest:

First, find out the record number of first log entry that match the date and time pattern using its regular expression (RegEx) function:

awk '{if ($1 ~ /2008-01-16-17/){print NR}}' < db2diag.log | head -1
Next, find out the record number of last log entry that match the date and time pattern:

awk '{if ($1 ~ /2008-01-16-17/){print NR}}' < db2diag.log | tail -1
Finally, use awk again to extract or filter all log entries within the range of first and last record numbers that we’ve known from last two steps:

awk '{if (NR >= 7529 && NR <= 8382){print $0}}' < db2diag.log
Because the nature of db2diag.log, the last record number I get from awk doesn’t include the detail of DB2 event / incident happened on that particular time. Thus, I purposely top up the “last record number” (suppose the last record number reported by awk command is 8382, I rest it to be 8390):

awk '{if (NR >= 7529 && NR <= 8390){print $0}}' < db2diag.log >tempfile
If you would like to output the extracted log entries to another temporarily file, just redirect the standard output of awk command to a temp file as you wish (e.g. append >tempfile to the end of last awk command sample).

Brief note about the awk programming syntax used in the sample codes at above:

$1 ~ /2008-01-16-17/ means to check if 1st field/column text pattern matches with the regular expression (i.e. 2008-01-16-17).

Unless the field separator (FS) is specified, awk regards space as field separator by default.

The first field (a.k.a column) of a line (awk treats each line as a record) is denoted as $1, 2nd field as $2, and so forth. The $0 is simply means all the fields/columns of the line/record.

Thus, the combination of awk programming and organized text files can form a simple database system!

The awk regular expression pattern is enclosed by a pair of slash character (/).

The awk RegEx operator for match comparison is a tilde/swung dash character (~). (refer to GNU awk notes on Regular Expression).

print NR is meant to print the record number (NR), i.e. the line number in the log file. To print the number of field/column in a line/record, use NF

WASHINGTON (Reuters) - President Barack Obama vowed Saturday to stick to the big-ticket items in his budget proposal but acknowledged that dollar amounts would "undoubtedly change" as Congress prepared to take up his record spending plan.

Trying to refocus attention from the AIG (nyse: AIG - news - people ) bonus scandal that has drawn public outrage, Obama stepped up defense of his $3.55 trillion budget for fiscal 2010, a linchpin of his efforts to rescue the ailing economy from the worst crisis in decades.

"It's an economic blueprint for our future, a vision of America where growth is not based on real estate bubbles or over-leveraged banks, but on a firm foundation of investments in energy, education and health care that will lead to a real and lasting prosperity," Obama said in his weekly radio address.

The budget committees of the Senate and House were set to begin crafting their budget legislation next week.

Republicans and even some of Obama's fellow Democrats who control Congress have complained that his budget, the first of his presidency, is too costly. It projects deficits of $1.75 trillion this fiscal year and $1.17 trillion next fiscal year.

Congressional budget experts Friday offered a darker economic and budget outlook, projecting a $1.8 trillion deficit this year which could complicate Obama's efforts to win passage of his 2010 budget.

Taking on his critics, Obama said: "These investments are not a wish list of priorities that I picked out of thin air.

"They are a central part of a comprehensive strategy to grow this economy by attacking the very problems that have dragged it down for too long: the high cost of health care and our dependence on foreign oil, our education deficit and our fiscal deficit."

Reminding listeners that he had inherited a "fiscal mess" from his Republican predecessor, George W. Bush, Obama -- who took office on Jan. 20 -- reiterated his pledge to cut the federal deficit in half by the end of his term.

But he acknowledged room for compromise on a final budget deal. "As the House and the Senate take up this budget next week, the specific details and dollar amounts in this budget will undoubtedly change," Obama said. "That's a normal and healthy part of the process.

He urged lawmakers to act with a sense of urgency, saying "the challenges we face are too large to ignore." (Editing by Chris Wilson)

Software is supposed to be a mature industry, characterized by some sort of mono- or duopoly. How to explain, then, the activity around Web browsers: Three of the tech industry's biggest names--Microsoft, Google (nasdaq: GOOG - news - people ) and Apple (nasdaq: AAPL - news - people )--each has a significant in-house browser development effort, with periodic fresh releases. Then, of course, there is the Mozilla Foundation, the folks behind the popular open-source browser Firefox.

Competition is always good, but especially these days with browsers. Features are being added to them that may, in a year or two, make a browser-based application look and feel no different from a desktop one. Imagine having the equivalent of Photoshop or PowerPoint in your browser--and thus available on whatever machine you happen to be using, desktop or smart phone or laptop.

The coming evolution in browsers is akin to the Ajax phenomenon of recent years. Ajax is a name given to a quartet of programming technologies that collectively made possible the likes of Google Maps and Gmail.

Before Ajax the typical Web site was a collection of static pages. With Ajax, programmers were able to change only part of the screen, displaying, for example, different information as you move a cursor around on a map. Ajax also allowed Web pages to be more dynamic in other ways, letting users, say, right-click and see a menu tailored to their needs.

The components of this new browser programming paradigm are esoteric. One is a new, extra-speedy Javascript interpreter, found in the latest browsers from Google and Apple, that allows programs in the browser's standard language to zip along faster than ever thought possible. Another is an Apple-created graphics technology known as Canvas, which gives programmers much more freedom using text and drawings.

Other under-the-cover changes include giving browsers the sort of sophisticated software-control features usually found only in operating systems. Web Workers, for instance, is an emerging system for isolating a browser's individual tasks into separate "threads," making it easier for a browser-based program to perform a computationally intensive task such as photo-editing in a background tab while the user is attending to something else, like e-mail, in the foreground.

This new approach to programming doesn't yet have a handy name like Ajax, though some refer to it as HTML 5. No browser yet has all of these new elements. Apple, Google and Mozilla have pieces. All are competing to add more.

So far there are no Google Maps-style killer apps for this new programming approach; indeed, programmers are just beginning to wake up to the possibilities. But one modest example is an early version of the sort of text editor used by engineers for writing computer programs. It's at bespin.mozilla.com and works with the Mozilla browser, Firefox.

Ben Galbraith and Dion Almaer, the Mozilla engineers who developed the site, said it will be expanding into a full-blown "programming environment" for the new software approach, but one that itself uses the same technologies that programmers will be making use of to build other applications. In a year or two, they say, software will be available that is indistinguishable from traditional desktop programs. The two men helped chronicle the Ajax movement; Galbraith said the new tools "have us more excited than we were for Ajax."

Many people assume that browser-based programs would run "in the cloud," that is, on servers situated remotely at companies like Google or Amazon. But Almaer said there's no reason software has to be written that way. A photo-editing program based in a browser, for instance, could run entirely on your desktop. PCs have power to spare.

Who wins and who loses with this new approach? Adobe (nasdaq: ADBE - news - people ) might not look too kindly upon it. The maker of Flash software would prefer that programmers stick with its software. Microsoft usually doesn't warm to standards it can't control; it is also pushing its new Silverlight multimedia program, which performs some of the functions of HTML 5 software.

Apple, Google and Mozilla, by contrast, favor anything that curbs Microsoft's market position. The three cooperate in browser development even as they compete for market share. If the day arrives when a browser is the only program anyone needs, those three would be among the ones cheering loudest.

Senior editor Lee Gomes covers technology from our Silicon

M.I.A.   
more photos ›

It was all about the mash-up at the 51st annual Grammy Awards: Justin Timberlake and Al Green and Keith Urban; the Jonas Brothers and Stevie Wonder; Radiohead and the USC Trojan Marching Band. Fashionwise, the story was much the same. To perform her prizewinning single, "Last Name," Carrie Underwood sported a getup that was Loretta Lynn up top, Madonna on the bottom. (Her quads, by the way, could give Madge a run for her money.) Accepting the first of three awards for the band, this one for Song of the Year, Coldplay's Will Champion gave a shout-out to Sir Paul McCartney, who was seated a few feet away: "Thank you, and sorry for blatantly recycling the Sergeant Pepper outfits—sorry, boss." And after walking the red carpet in a Bollywood-bright Manish Arora dress with sneakers, mommy-to-be M.I.A. took the stage wearing a sheer House of Holland outfit that showed off her full-term belly in all its glory. Backing her up was the "Rap Pack": Kanye West, Jay-Z, T.I., and Lil Wayne. But the pairing that will really have the fashion world talking? That would be Best New Artist winner Adele and her date, Vogue's Hamish Bowles.

We take a quick look at 14 of the best 400 firms in America.

They stand in line outside stores waiting for midnight launches of new video games. When they get home after a long day, they plop down in front of the TV not to sit back and watch, but to play.

They're known as "core gamers." They are people like Greg Wilcox, who writes about video games and has bought roughly 100 this year, and people like Mark Hengst, who's in law enforcement and says daily gaming gives him an "interactive form of escapism." And there's Wyatt Du Frane, a geology graduate student who's been playing since he was a little boy.

"I like their scope," said Du Frane, 28, a student at Arizona State University. "A movie is only a couple of hours. A video game is more like a book or a TV series, where you can kind of continue the story."

For the video game industry, core gamers are proving crucial. Their willingness to regularly, loyally buy new titles - no matter what - gives the industry a better chance of success than other businesses that rely on discretionary spending vulnerable to the recession.

"As long as hard-core gamers have a job, they will continue to buy games," said IDC video games analyst Billy Pidgeon.

The industry's ability to lean on core gamers is a bit of a twist, because video game makers have been working hard to grow by expanding their mainstream appeal.

Families and people who haven't picked up a game controller in ages, or ever, have flocked to the easy-to-master Nintendo (other-otc: NTDOY.PK - news - people ) Wii since its 2006 launch. Taking note, Sony Corp. (nyse: SNE - news - people ) and Microsoft Corp. (nasdaq: MSFT - news - people ) have been expanding what their game consoles offer, adding movies and TV shows, to attract people whose idea of the perfect Sunday afternoon doesn't involve shooting aliens. Software publishers like Ubisoft Entertainment, Activision Blizzard (nasdaq: ATVI - news - people ) Inc. and Electronic Arts Inc. (nasdaq: ERTS - news - people ) have boosted their titles aimed at young girls, families and women over 35, who have helped push game sales higher.

Ford Motor is suffering from guilt by association. The automaker has $15 billion or so in the bank and billions more in credit lines, is not looking for a year-end bailout and still gets splashed with mud. Every day I hear the TV news people, the stars like CNBC's Maria Bartiromo, lump General Motors, Ford and Chrysler together as facing bankruptcy. In Ford's case, this is just not true.

Alan Mulally, the chief executive Ford imported from Boeing (nyse: BA - news - people ), has moved smartly since he gave up his wings. He mortgaged assets (for $24 billion) and signed up credit lines two years ago before all the current turbulence.

He also sold Jaguar, Land Rover, Aston Martin, some of Ford's Mazda (other-otc: MZDAF.PK - news - people ) stake and has put Volvo on the block. You can disagree or agree, maybe some of these operations could still have turned into winning assets, but Mulally decided Ford did not need the problems right now.

Unlike GM, Ford has no surplus car lines, which means it avoids both heavy spending to keep too big a lineup up-to-date and endless lectures from Wall Street know-it-alls who say to get rid of them. Excluding Volvo, which Ford hopes to sell, and Mazda Motor, in which it has only a minority stake, Ford has only two dealership channels in this country: Ford and Lincoln/Mercury.

Both Ford and GM are unlike Chrysler in that they have robust foreign operations. For Ford, Europe and South America earned $2.5 billion pre-tax in the first nine months this year. Those markets are slowing, yes, but they are strong businesses. Europe is providing the small-car knowledge and engineering that Ford needs in the U.S.

Yes, Ford has asked for a government-backed credit line, just in case the economic downturn gets much uglier, and is asking for some of the government cash that Congress already appropriated for updating plants and making fuel-saving vehicles. On the other hand, Ford is not begging for an immediate cash infusion to keep it afloat.

Long run, Ford has the ability to grow. For the past two months the Dearborn, Mich., manufacturer has held its own in share against the prior year, while the others slipped. The company even picked up share in November, to 16.4% of the industry sales versus 15.4% a year before. This is a good sign. If GM downsizes, Ford could end up bigger than GM in just a few years.

How Unions Stop The Cars

Big Labor is a big problem for automakers' survival.

With the late-night demise of legislation containing $14 billion in emergency loans to Detroit's automakers, pressure is once again mounting on President Bush to step in. And he is reportedly thinking of doing just that. But the very thing that doomed this legislation will also doom any effort to rescue the industry: union intransigence. If Bush cares more about taxpayers than kudos, he should decline.

The legislation, backed by Sen. Bob Corker, a Tennessee Republican whose state itself is home to GM facilities, was the industry's best hope to return to health. It stripped some of the green baggage of the House bill that would have consigned Detroit to producing not cars that sell but what eco-warriors want. Nor would the legislation have handed quite as expansive powers of micromanagement to a car czar, forcing companies to obtain approval for basic product and capacity decisions.

Instead, it offered the automakers a way to restructure their massive obligations to labor and debtors, much like a bankruptcy court would do but without the stigma. Bondholders would have been required to accept a 70% loss--the remainder paid in stock, not cash. And Big Labor's main concession (besides accepting some stock instead of cash for its health care trust fund) was that it set a definite date for a pay cut next year.

At that time, its wages and benefits would fall in line with those that Nissan (nasdaq: NSANY - news - people ), Toyota (nyse: TM - news - people ) and other automakers pay their U.S. workers.

But the United Auto Workers reacted as if it had been asked to work in a Third World sweat shop and walked away. Sen. Debbie Stabenow, D-Mich., decried efforts to "sock it" to American workers. Never mind that labor costs make every car rolling out of Detroit $1,500 more expensive to produce than foreign cars made elsewhere in the U.S. Indeed, last year, GM and Toyota sold the same number of cars worldwide, but Toyota turned a healthy profit--while GM posted a $40 billion loss.

But the fact of the matter is that the wage cuts are a necessary condition to give Detroit a fighting chance for survival, but they're not sufficient. Indeed, that would require far more from unions.

Car sales next year are expected to drop 40%. This means that if auto companies are going to use any bailout money to restore viability, they will have to be able to shed some of its quarter-million-strong workforce.

However, if the UAW was unwilling to accept a pay cut, there is no reason to believe that it would compliantly accept such massive layoffs. More likely, it will use taxpayer money to keep every job alive as long as possible--and then return for more a few months later.

Beyond job cuts, the UAW will also have to agree to eliminate a whole host of exceedingly rigid work rules for its remaining constituents. Such rules, for instance, had historically made it difficult to train auto workers for multiple jobs to fulfill multiple needs. No less than labor's extravagant wage demands, these rules have crimped Detroit's adaptability.

Ford recently built a facility in Brazil where it can produce five different vehicle platforms at the same time, on the same line. What's more, many of its suppliers are housed in the facility as well, something that allows them to move parts to the assembly line at a moment's notice. Not only has this lowered Ford's production costs and boosted productivity, it has also given it flexibility to adjust its product mix to shifting market conditions. This is important at any time but is especially crucial now, when volatile oil prices are likely to produce abrupt shifts in consumer demand.

But union rules, with their featherbedding requirements and crabbed job descriptions, make it much harder for such a factory-of-the-future to operate in the U.S.

The irony is that foreign car makers are profitable in America--and the Detroit Three are profitable in every country but America. Only Big Labor can position Detroit carmakers for success in their own country. Bush shouldn't ask already-strapped taxpayers to make sacrifices to pull Detroit back from the precipice when its own key stakeholder won't.

Shikha Dalmia is a senior analyst at the Los Angeles-based Reason Foundation. She can be reached at shikha.dalmia@reason.org.

BCE Inc. said Friday that it plans to buy back up to 5 percent of its common shares and resume paying dividends following the collapse of the deal to take Canada's largest telecom company private in what would have been the biggest leveraged buyout in history.

The parent company of Bell Canada said it will repurchase up to 40 million outstanding common shares and will reinstate its quarterly dividend at 36.5 Canadian cents per share.

"A share buyback is the most efficient method of distributing capital to our shareholders, particularly given the current valuation metrics of the company," said chief financial officer Siim Vanaselja.

The buyback would cost BCE about 840.8 million Canadian dollars ($677 million) at its price at midday Friday.

BCE said earlier this week that it would restore the dividend and buy back stock following the collapse of the proposed $35 billion buyout by an investor group led by the Ontario Teachers' Pension Plan and several U.S. partners. The investors group had expected to complete its deal for BCE on Dec. 11.

But the deal fell through after a review by accounting firm KPMG found it would have left the company in violation of solvency tests of the privatization agreement, partly due to the amount of debt involved in the transaction and current market conditions.

There were also arguments over a breakup fee. The buyers group had said that no breakup fee will be paid, but BCE said in a separate statement it will demand payment of 1.2 billion Canadian dollars ($970 million).

Bell Canada issued a statement Friday saying that it will continue to move forward as a re-energized company and is supportive of BCE's buyback plans.

"Given this steadily improving business trajectory, we view the dividend and share buyback initiatives announced by BCE today as very attractive to our shareholders now and going forward," said George Cope, president and CEO of Bell and BCE.

BCE said the first new dividend payment will be made Jan. 15 to shareholders of record on Dec. 23. BCE also scheduled its annual meeting of shareholders on Feb. 17 in Montreal.

The dividend yields 6.95 percent at Friday morning's share price of 21.02 Canadian dollars, down CA$1.01 in trading in Toronto.

That share price is down from CA$38.35 just before it became apparent on Nov. 26 that Teachers' cash bid of CA$42.75 a share would not proceed.

The Toronto-based Ontario Teachers' Pension Plan -- with assets of CA$108 billion ($87 billion) in 2007 -- invests and administers the retirement funds for Ontario's 353,000 active, inactive, and retired teachers. U.S.-based Providence Equity Partners and Madison Dearborn Partners LLC are also involved in the proposed buyout.

BCE, which has more than 54,000 employees, had annual revenue of CA$17.8 billion ($14.4 billion) in 2007. It had 5.8 million wireless subscribers, 8.64 million phone lines, 1.94 million Internet subscribers and 1.82 million satellite television subscribers in 2006. It is Canada's largest communications company.