Pakistani President Asif Ali Zardari insisted his government was safe hours before Wednesday's White House talks with Afghanistan on how to fight a raging Taliban insurgency. The summit, one of President Barack Obama's most high-profile diplomatic engagements yet, comes as US lawmakers voice increasing fears that Pakistan is losing the fight against Islamic extremists. The government was preparing to take in some 500,000 people who might seek refuge from Pakistan's Swat Valley, a one-time ski resort near the capital Islamabad where the military has launched an offensive after a controversial cease-fire with militants. Officials in Pakistan said Wednesday that more than 40,000 people were fleeing the fighting in the country's northwest. Zardari cast the Taliban insurgency as another chapter in Pakistan's long-running tribal and ethnic conflicts. "They're not threats to my government. They are threats to my security," Zardari told CNN. "My government is not going to fall when one mountain is taken by one group or the other," Zardari said. Zardari -- whose wife Benazir Bhutto was assassinated in 2007 -- also said his nation's nuclear arsenal was in secure hands. But some US lawmakers preparing a giant aid package for Pakistan voiced doubts about Zardari, a civilian who took over last year after nearly a decade of military rule. "Let me blunt -- Pakistan's pants are on fire," Democratic Representative Gary Ackerman said. "The fire is real, and they need to respond." "There is a real and present danger to Pakistan's survival, but it comes from inside, not outside the country," he said, faulting the Pakistani military's focus on historic rival India. Obama has placed Pakistan at the center of the fight against the Taliban's Al-Qaeda allies as he dispatches 4,000 more troops to Afghanistan, in addition to an extra 17,000 already committed. Two US senators have introduced legislation that would triple US civilian aid to Pakistan to 1.5 billion dollars per year over the next five years. Richard Holbrooke, the US special envoy to the region, pledged strong support for Zardari, denying media reports that the United States was reaching out to his arch-rival Nawaz Sharif. "Pakistan's of such immense importance to the United States, strategically and politically, that our goal must be unambiguously to support and help stabilize a democratic Pakistan headed by its elected president, Asif Ali Zardari," Holbrooke told Congress. But he stressed that security assistance for Pakistan "has to show results.""Pakistan must demonstrate its commitment to rooting out Al-Qaeda and the violent extremists within its borders," Holbrooke said. Zardari later held a 90-minute closed-door meeting trying to "reassure" members of the House Foreign Affairs Committee on Pakistan's commitment, said Democratic Representative Gerry Connolly. Pakistan has resented attempts by lawmakers to impose benchmarks in exchange for US aid, saying they were politically unfeasible considering the public's deep distrust of US intentions. Holbrooke, speaking later to PBS public television, showed sympathy to the Pakistani position. Holbrooke said that while it was "perfectly rational" to set benchmarks for US aid, "if you attach too many conditions you can end up with a disastrous outcome in which the very thing you're trying to do is undermined." Afghan leader Hamid Karzai, addressing Washington's Brookings Institution, said that the White House summit would discuss closing militant "sanctuaries" in Pakistan. Afghanistan's relations have improved with Pakistan since Zardari took over, but Afghan and US officials charge that rogue elements in Pakistan's military and intelligence services are supporting Taliban and Al-Qaeda extremists. "The return of the Taliban is because we did not address the question of sanctuaries in time. Unfortunately, today, Pakistan is suffering with us massively as a consequence of that," Karzai said. The Afghan and Pakistani leaders are also expected to raise their concerns about the deaths of civilians in military operations, warning that they alienate the public. "This war against terrorism will succeed only if we fight it from a higher platform of morality," Karzai said. "Money can't buy you love, as you say it in America, no matter how much it is." Pakistan was the main backer of Afghanistan's hardline Taliban regime until the September 11, 2001 attacks on the United States, which then invaded and paved the way for Karzai to take office in Kabul.