The former head of Microsoft’s search team has created a new search engine, CrowdEye, to help people sort through Twitter and find what’s popular.

Twitter’s trending topics are a helpful place to start if you want to see what people are talking about on the Web. The problem is that there are some many tweets under those trending topics that it is hard to make sense of it all. A popular topic could have hundreds of tweets coming in every few seconds, and that’s just too much data to try to process.

CrowdEye hopes to alleviate this problem. It was just released into beta and has some bugs, but it is a very useful tool for certain situations and should hopefully continue to grow in the coming months. Here is what CrowdEye’s about page has to say about the new service:

By tracking discussions on Twitter, we can help our users find out what’s important to them right now in real time. CrowdEye has created innovative technology to scan through tweets, retweets, twitter links and more. We then provide you with powerful yet easy ways to slice, dice, summarize and categorize the data to answer your questions.

CrowdEye is especially helpful for finding relevant content and context about subjects. For instance, #IranElection has been the top trending top on Twitter for days, but it is filled with so many tweets that it is hard to follow. CrowdEye will show you the most tweeted links on a particular topic over the past 12 hours, day, two days or three days.

In addition, CrowdEye also breaks down hastags like #IranElection into popular topics like Mousavi, fraud, protests, Ahmadinejad, etc. If you want to see popular links and recent tweets under the #IranElection hashtag that mention protests, CrowdEye makes that easy.

Even cooler is that CrowdEye provides unique tags for each topic under a hashtag. Some popular tags for Mousavi are logo, march, rally, opposition, etc. Some Popular tags for Ahmadinejad are Photoshop, rejection, fear, hardliners, diverse, etc.

It gets even better. I can then dial in and find popular links being shared under the #IranElection hashtag that mention Ahmadinejad and Photoshop. The image below shows the most popular links in the past day that talk about the Photoshop controversy surrounding President Ahmadinejad:

Not everything is rosy. The recent tweets are fairly worthless and show no clear weighting right now. The service was just launched in beta and this may improve in the future, but as of right now, it’s not nearly as useful as tracking which links are being shared.

CrowdEye is off to a strong start and could really be a great tool for people trying to make sense of it all on Twitter. The service is a bit rough right now, but it’s still quite useful.

It’s true that the Web, smartphones, social media, blogs, etc are making it easy for everyone to report and share their stories, but that doesn’t mean that there isn’t a need for professional journalists. In fact, I’d argue that citizen journalism, while helping to cover the world better, only highlights the need for professional journalists.

Over the past few days, I’ve extolled the virtues of social media in covering the unrest in Iran. True, without social media, this story might not be told properly, but there still has been a large need for professional reporters. Let’s take a look at some content from pros that has really helped provide clarity to what is going on in Iran:

FiveThirtyEight.com — The new media startup/blog that covered last year’s presidential election so well has a great piece on the suspect numbers coming out of Iran. FiveThirtyEight made its name by analyzing polling data, and in a post today it compares the 2005 Iranian presidential election to the one last week.

This is the kind of thoughtful, time consuming analysis that can’t be provided in 140 characters. It’s also the kind of analysis that can help make a journalism organization stand out.

Some irregularities that popped out:

Around 1600 GMT Sunday, the ministry of Interior released the official vote totals by province. As others have mentioned, by law candidates have three days following voting to contest the result, before the final totals are approved by the Supreme Leader. As such, it is notable that both the aggregate totals and provincial totals were certified, approved and released before the three day deadline.

We would have expected Ahmadinejad’s result from Friday, informed by the polling, historical trends and a bit of bet-hedging, to be between 40% and 55%. These figures would suggest that Ahmadinejad’s reported 65% of the national vote is at minimum outside of the trend, and more likely, an exaggerated figure.

Medhi Karroubi, over whom Ahmadinejad advanced to the 2005 runoff round by just 700,000 votes, was surrounded by controversy in that election as well, arguing that Ahamdinejad’s totals had been inflated by conservative hardliners. His openly accusatory allegations to the Supreme Leader resulted in his resignation from several top political posts.

This post is a must read for anyone who wants to understand why the election results are so suspect.

The Lede | The New York Times — The Lede has been leading the NYT’s coverage of Iran by curating the best content from around the world about the Iranian elections. The Lede is linking to other news organizations, bloggers, press reports, videos and, of course, New York Times content.

The Times provides an excellent example of how new and old forms of journalism can merge together to cover a story better. The Times still has its excellent traditional news stories that help put everything into context. These are the kinds of stories that are needed to help people make sense of all the upheaval in Iran.

But the Times also has The Lede, which is focusing on curating right now to provide the total picture of Iran. Professional journalists make excellent curators. Many are quite knowledgeable on certain subjects and make ideal people to curate content from around the Web.

The Iran election aftermath cannot be told fully by one news organization. That’s why a a mixed strategy of original reporting and curation of the best of the rest makes sense. The Lede is the ideal place to start on nytimes.com when looking for coverage of Iran.

BBC News — BBCNews.com has an excellent who’s who in Iran post up that makes an excellent primer for anyone wanting to know the major players in Iran. This is the kind of simple, yet information piece that you won’t see originating on social media. It provides excellent context for what’s going on and invites readers — especially non-Iranians — to learn about the key players.

Analysis content is where professional journalists can really stand out. On this front, the BBC has several great pieces on BBCNews.com. They have a Q&A with their correspondent in Tehran and an analysis piece comparing previous mass Iranian protests to the new ones.

Citizen journalism is here to stay, and it’s going to help provide fuller coverage of the world. But there is still an important place for professional journalists, especially when it comes to putting everything into context.

These ground details are even more important now that the Iranian government is blocking foreign journalists from reporting on rallies. While the Iranian government may have success blocking foreign journalists from covering rallies, it has no hope of containing the legions of people with mobile phones that can send tweets, photos and videos to the Web.

Here are some tweets from today:

“CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT”

“Basij taking cameras. I fear a massive crackdown. Don’t let them into the protest.”

“Try to peacefully keep Army and Basij from keeping formations around the protest. Don’t let them surround you! #iranelection”

“demo today v\peaceful but it is going to change tonight – too many baseej tody #Iranelection”

For those of you who don’t know, Baseej or Basij are decentralized militia that have branches in about every Mosque in Iran. It was created during the 1979 Islamic Revolution in Iran. Many of the basij are young boys (with guns) and have been accused of human rights abuses.

The Basij underwent a revival after Mahmoud Ahmadinejad was elected president. So when someone on Twitter says that demonstrations will no longer be peaceful because of Baseej, it means that young, armed boys are coming out to dispense their own brand of justice in the name of Ahmadinejad.

There have already been several protestors killed because of the Basij. The Iranian government can try all it wants to censor foreign journalists, but it has no hope of stopping every Iranian from reporting with their cell phones.

Check out our post from yesterday on social media’s impact on what is happening right now in Iran.

It’s starting to become a little ridiculous for people — many of them old-school journalists — to deny the power of Twitter, especially in light of what is happening in Iran right now.

I can offer no great insight into who really won the recent presidential election, but it is clear to everyone that many people in Iran are not happy and feel they have been screwed over. Again, however, it is social media leading the way for coverage. If you want to know what’s really going on in Iran, Twitter is the place to be.

Right now, four of the top trending topics on the service are IranElection (No. 1 right now), Tehran, Iranians and TwitterReschedules. The last topic is about how Twitter has rescheduled routine maintenance, as not to disrupt the current chatter about Iran. It’s a good thing that Twitter at least recognizes the seriousness of this situation, because many in at traditional media outlets haven’t paid much attention to this unfolding story.

The Iranian government controls the media. The BBC even believes that Iran is responsible for their satellite signal being blocked in the region. That’s exactly where a subversive social media technology like Twitter comes in.

Decades ago, a totalitarian government could have made it extremely difficult for the outside world to know what was happening in their country. To this day, North Korea and its internal workings are shrouded in mystery. State-run media can silence dissidents internally as well.

But no government has found a way to silence the broader Web. Sure China tries, but even the great firewall has great cracks. In Iran, we are seeing that the Web — and more specifically social media — cannot be silenced.

What makes Twitter such a subversive tool is that it is so hard to block and stop. Anyone with a mobile phone (and there are many more mobile phones than computers in the world) can post to Twitter. Those with more advanced phones can upload photos and text via the service. It’s difficult to stop millions of people from sharing their experiences via a network like Twitter.

Sure, Iran is trying to block all this information from coming out, but that’s far easier said than done. On the Web, even if the government finds a way to block a Web service, it won’t stay blocked for long. Alternative proxies and other workarounds quickly propagate throughout the Internet.

Beyond the merely technically is the shear scale of the problem for the Iranian government. There are lots of social media sites out there to try to block, and even if the government managed to block all of them, it would still have to contend with millions of blogs. The beauty of the Web is that it allows anyone’s voice to be heard.

For the real story, one needs to be on Twitter, YouTube, Flickr, etc. The mainstream media like CNN have been woefully inadequate in covering the turmoil (although they have been coming around). In fact, for most of the weekend, CNN.com did not feel the unrest in Iran was worthy of being the top story.

Ironically, it was Twitter users who slammed CNN the hardest. #CNNFail become a popular hashtag on the service as angry users slammed the network for taking such a caviler approach to this issue. Ever since then, CNN.com has taken a far more serious and in-depth approach to the upheaval in Iran.

Here is what some Iranians are saying on Twitter:

“I’ve learned something today. Americans DO care about the world outside America. Their media just doesn’t.”

“Non stop sound of shooting heard in Tehran.”

“Just saw pics of dead bodies. Bodies of young iranians. Got sick and cried for hrs.”

“Good night. viva freedom. viva truth. Hope a better coverage by media. That’s our only support.”

“I’m so tired and going to get some rest while I know there r people & students in streets fighting for justice.”

YouTube big in coverage too

Not only are people sharing powerful images and text via Twitter, but a myriad of user-generated video is appearing on YouTube. These are the kinds of video that traditional media outlets rarely get. These are also the kinds of video that the MSM may hide from the public because they are too raw (read: to real).

For instance, take this video of a crowd of protestors being shot at:

Here is another video of protestors chanting into the night:

Professional journalism bringing analysis, insight & context

There is, of course, room for professional journalists in this equation. Professional reporters can make sense of all these tweets, photos and videos. Professional reporters can also offer additional on-the-ground insight.

Beyond that, professional journalists can offer analysis and try to answer the why question. The ideal future of media involves a collaboration between citizen and professional journalists.

These are the kinds of stories you won’t find originating on Twitter:

• Iran’s Supreme Leader Ali Khamenei has ordered an investigation into charges of voter fraud
• BBC Iranian affairs analyst Sadeq Saba looks at the key questions in the wake of the county’s bitterly contested presidential election result
• In Iran, an iron cleric, now blinking
• ANALYSIS – No-win situation for Obama team on Iran

All of these photos come via Flickr, another social networking site that is helping to spur this revolution.