Imam Feisal Abdul Rauf, who is leading efforts to build a mosque near Ground Zero, the site of 9/11 attacks in New York, has warned that retreating on the project would only strengthen the hand of Muslim extremists around the world. At the same time, the Imam appeared to deflect a question when pointedly asked if he intended to keep the Islamic cultural center at its current site, two blocks from where extremists crashed planes into the World Trade Center. "The decisions that I will make -- that we will make -- will be predicated on what is best for everybody," he told ABC's "This Week" programme on Sunday. Further asked by Christiane Amanpour, the anchor of the programme, if he would have proposed the Islamic Center project knowing the controversy it would spark, Imram Rauf responded, "I would never have done it. I'm a man of peace. The whole objective of peace work is not to do something that would provoke controversy." However, things have gone so far and his proposed Islamic community center, has gained so much attention, moving it to another location or canceling the project isn't quite so simple, he said. 9/11 RALLIES On Saturday's ninth anniversary of the 9/11 attacks, thousands marched in dueling protests over the row, which was stoked by a Florida pastor's threat to burn the holy Quran mark the occasion. The rallies were mostly peaceful, although there were heated arguments between the two sides. Pastor Terry Jones used his threat, which triggered demonstrations across the Muslim world, as a bargaining chip to try and get the mosque moved and flew to New York to meet the imam. Abdul Rauf has so far refused to meet him and his initial response in a statement to CNN was: "We are not going to toy with our religion or any other. Nor are we going to barter." The imam told ABC in the interview on Sunday that the "discourse has been, to a certain extent, hijacked by the radicals," and said this made his decision on the mosque "very difficult and very challenging. "The radicals on both sides, the radicals in the United States and the radicals in the Muslim world, feed off each other. And to a certain extent, the attention that they've been able to get by the media has even aggravated the problem." The mosque, to be built on the site of a derelict clothing store two blocks from Ground Zero -- the name given to the site of the downed Twin Towers -- was proposed by Rauf as a way of giving Islam a new face in the United States. "My major concern with moving it is that the headline in the Muslim world will be Islam is under attack in America," the imam told ABC. "This will strengthen the radicals in the Muslim world, help their recruitment," he continued. "This will put our people -- our soldiers, our troops, our embassies, our citizens -- under attack in the Muslim world and we have expanded and given and fueled terrorism." New York Mayor Michael Bloomberg has warmly endorsed the mosque idea and President Barack Obama has repeatedly said that religious freedoms in America allow anyone to build a holy site where they please. Polls show a majority of Americans, and some 70 percent of New Yorkers, think the mosque should be moved further from Ground Zero, not least because of the sensitivities of those who lost loved-ones in the 9/11 attacks. Opposition to the mosque has developed into a cause celebre for ultra-conservative radio hosts and Tea Party movement activists dedicated to battling Obama and what they denounce as his liberal, "socialist" agenda. The row was further inflamed by pastor Terry Jones and the threat of his Dove World Outreach Center -- a tiny church in the town of Gainesville with a congregation of less than 50 -- to burn 200 copies of holy Quran on Saturday. Abdul Rauf said that if the Quran burning plan had been implemented, it would have been "a disaster" in the Muslim world. "It would have strengthened the radicals," he said. "It would have enhanced the possibility of terrorist acts against America and American interests." The imam described a "growing Islamophobia" in America, but said that contrary to the claims of radical Islam, Muslims in the United States were free to observe their religion, happy and thriving. Abdul Rauf said the way America was treating its Muslims was being watched by over a billion Muslims worldwide and that the main ideological battleground today was not between Islam and the West. "The battleground has been: moderates of all faith traditions in all the countries of the world against the radicals of all faith traditions in all parts of the world." The proceedings climaxed midafternoon when an overwhelmingly anti-mosque crowd jammed two blocks along West Broadway for a rally hosted by the organization "Stop Islamization of America", a Jewish-backed group. Event organizers trotted out a procession of speakers -- including family and friends of 9/11 victims, the Dutch politician Geert Wilders and former U.S. Ambassador to the United Nations John Bolton on videoon a stage erected just a few yards from the site of the planned Islamic center. What started as a tribute to the victims of the 2001 attacks evolved into a stream of anti-Islam rhetoric that fueled the crowd. Wilders, the leader of the Dutch anti-Islamic Freedom Party, drew some of the loudest cheers as he questioned the motives of Imam Rauf. Speakers were regularly were interrupted by chants of "No more mosque" which at times they led. Members of the packed crowd waved American flags and held signs that included "Muhammed (Peace be upon him) was the first radical Muslim. Osama Bin Laden followed his direction." Opponents and supporters of the project began trickling into the streets of lower Manhattan early on Saturday. One of the supporters, who carried a sign saying "The Attack on Islam is Racism," read from the Quran while the Vietnam veteran, 61-year-old Ed Dougherty, recited the Constitution. A man wearing a New York Fire Department uniform shouted "Go talk about Christianity in Pakistan" President Barack Obama told a deeply polarized America that Islam is not the enemy as somber ceremonies marked an unusually tense ninth anniversary of the September 11 attacks. "As Americans, we will not and never will be at war with Islam. It was not a religion that attacked us that September day. It was Al-Qaeda, a sorry band of men, which perverts religion," he said. Imam Rauf also said he believes politicians and pundits have poisoned the process. "What has happened is that since May... certain politicians decided that this project would be very useful for their political ambitions," he said, noting that people like Sarah Palin are fueling a "growing Islamophobia" in the United States. Palin posted via Twitter in July: "Peace-seeking Muslims, pls understand, Ground Zero mosque is UNNECESSARY provocation; it stabs hearts," she tweeted. "Pls reject it in interest of healing." "I felt it disingenuous," Rauf said of Palin's remark. She was later joined by politicians such as Newt Gingrich and Senate Majority Leader Harry Reid, who are not from New York City, but injected themselves into the Islamic center controversy to criticize the plan. "We are Americans, too," the Imam said. "We are doctors. We are investment bankers. We are taxi drivers. We are store keepers. We are lawyers. We are -- we are part of the fabric of America." Rauf fears that, with so much tension now, moving the mosque project is problematic and potentially dangerous. "Radicals" in the Muslim world have also made further discussion regarding the center more difficult. "We have two audiences," he said on "This Week." "We have the American audience and we have the Muslim audience. And this issue has riveted the attention of the whole Muslim world." Rauf said he was aware of polls that indicated a majority of New Yorkers wanted the mosque moved, but noted that any decision regarding the future of the Islamic center must be weighed carefully. "Whatever we do and whatever [we] say and how we move and the discourse about it is being watched very, very closely," he said. "If we make the wrong move, it will only expand and strengthen the voice of the radicals and the extremists."