TOKYO—Japan's prime minister warned China that its inflammatory reaction to a territorial dispute—from violent protests to apparent informal trade sanctions—could further weaken China's already-fragile economy by scaring away foreign investors. The comments showed the risks that the diplomatic standoff could broaden into a damaging commercial tit-for-tat between the world's second- and third-largest economies.

"China should be developing through the various foreign investments it receives," Yoshihiko Noda told The Wall Street Journal following a tense week filled with news of Japanese factories torched and cars overturned, and Chinese patrol boats hovering in and around territorial waters controlled by Japan. "I hope for its level-headed and rational understanding that anything to discourage that is a disservice to itself," the prime minister added during the interview in his residence Saturday.

"Recent delays in customs and visa issuance are of concern," Mr. Noda continued, referring to reports that Japanese companies were now facing a form of economic harassment, as he offered his most detailed comments to date on the business implications of the row. "Damaging our ties over such things would be bad for not just the two countries' economies, but for the global economy," he added.

Japan is simultaneously ensnared in an increasingly bitter tiff with another neighbor, South Korea, both over a separate territorial argument, as well as a debate over whether Japan has made adequate amends for its World War II aggression. Mr. Noda made clear in the interview that his government had no intention of making the concessions Seoul has demanded as necessary for repairing diplomatic ties frayed in recent months, indicating an extended period of friction there as well.

Asked if he would consider providing new compensation for the so-called comfort women who served as sex slaves for the Japanese soldiers, Mr. Noda said firmly: "The matter is closed." He said South Korean criticism that Japan's previous offerings were insufficient "hurt the feelings of conscientious Japanese, and it is a pity."

Mr. Noda conducted the interview in advance of a visit to the United Nations General Assembly session starting Monday, where he plans to give a speech highlighting the importance of "the rule of law" in international dispute settlement. While aides said that is intended as an allusion to Japan's tiffs with its neighbors, Mr. Noda said he wouldn't single out any countries by name: "I don't think it's appropriate for a leader to go on in length and in detail about individual issues."

Indeed, at the same time that Japan appears to be taking a firm line in its regional disputes, officials are also looking for ways to dial down the heat, particularly with China. Illustrating his balancing act, Mr. Noda held out hope that the recent curbing of formal diplomatic contacts would be eased, and that the foreign ministers of the Asian giants would meet in New York as a step toward defusing the tensions over the contested islets in the East China Sea. "If there is a chance, we should hold such a meeting," he said.

But Beijing indicated Sunday that it wasn't ready to move on, with the official Xinhua news agency reporting that the Chinese government had decided to cancel various ceremonies scheduled for later this week related to the 40th anniversary of normalization of postwar diplomatic ties between the two nations.

The latest flare-up with China followed the Japanese government's purchase from private Japanese owners earlier this month of the islands controlled by Japan but also claimed by China. Japan's nationalization of the islands—known as the Senkaku in Japan, Diaoyu in China—set off anti-Japanese demonstrations in more than 100 cities in China, some violent, leaving a trail of burned-down factories, looted stores and vandalized consulates. The head of a Japanese insurers' industry body has said insurance payments to Japanese companies damaged by the protests could total up to tens of billions of yen.

Beijing also sent paramilitary patrol boats under its various agencies to waters near the islands, setting off a cat-and-mouse chase with the Japanese Coast Guard that has continued for more than a week. Japanese opposition lawmakers have called for beefing up the defenses around the islands. But Mr. Noda said he saw no need to bring in the Japanese navy, and shrugged off the possibility of a military conflict. "It's not good to talk about pessimistic scenarios," the 55-year-old leader said.

Mr. Noda also gave Chinese officials some credit for trying to calm things down, saying "the government itself is calling for restraint, and I believe it is trying to defuse such behavior and actions," he said, referring to the protests. "However, I don't think they've been entirely successful yet," he added. And he noted that while the violence had died down, he cited the new trade delays as evidence that prospects remain for spreading economic fallout. In recent days, Japanese companies have reported customs-clearing delays, raising concerns that China was now retaliating economically, just as it did after a similar territorial spat in 2010 led to a curb on the export to Japan of rare-earth minerals vital for the production of some of Japan's leading products such as hybrid cars.

Mr. Noda implied that other countries would conclude they are vulnerable to the same kind of harassment facing the Japanese—and possibly curb investments—making a point of citing the incident in Beijing on Tuesday where anti-Japanese protesters briefly surrounded the American ambassador's car, causing minor damage. "Even the U.S. embassy and its official car came under attack," Mr. Noda said.

Mr. Noda's remarks Saturday followed statements by Japanese business leaders that they may begin to rethink their ties to their giant neighbor. Tokyo has considerable economic clout: Japan has become a major contributor to China's growth, just as China has become economically vital to Japan. Japanese companies injected $12 billion in foreign direct investment into China last year, according to Japanese government statistics. Japan is China's second largest trading partner, while China is Japan's largest.

While more understated than the rhetoric coming out of China, Mr. Noda's comments over the weekend echoed remarks by Chinese government officials and editorials suggesting that tensions could spill into business ties. The China Daily, for example, carried a column titled "Consider Sanctions on Japan." "Japan's economy will suffer severely if China were to impose sanctions on it. China's loss would be relatively less," said the piece written by an analyst at a think tank affiliated with China's Ministry of Commerce.

In addition to the isles dispute with China, Japan faces another with South Korea, which has evolved into an issue over Japan's wartime compensation. The South Korean foreign minister is widely expected to raise demands for new "comfort women" compensation in his U.N. speech. The long-simmering issue has heated up over the past year, after the South Korean constitutional court ruled that the country's leaders had violated the law by failing to negotiate a new compensation package with Japan. That put new pressure on South Korea's leaders, and Seoul has since twice asked Tokyo to hold consultations, but the requests were turned down.

South Korean President Lee Myung-bak cited the lack of progress on the issue as a reason for the recent flare-up in their bilateral territorial rivalry over a group of tiny islets known as the Liancourt Rocks, including his surprise August visit to the area called Dokdo in Korea and Takeshima in Japan. After that, South Korea's National Assembly passed a resolution demanding more formal apologies and compensation from the Japanese government over the comfort women issue.

In contrast to his remarks about China, which appeared to leave the door open for discussions, Mr. Noda was resolute Saturday in refusing Seoul's demand for government compensation, saying that South Korea agreed to forfeit claims to wartime compensation when the two nations normalized ties in 1965.

"In the long negotiation process, we have completely and legally resolved the matter of compensation," Mr. Noda said. He said the government went out of its way to jointly set up a fund in 1995 with private donations to provide the former comfort women with compensation. South Korea has said the fund should come directly from Japan's government. Mr. Noda said Korea should first revise its assessment of the fund that Japan has already offered before the two sides can look into alternatives.

Facing some criticism at home that Japan hasn't done enough to explain globally its side of its disputes with South Korea, Mr. Noda said: "We've been conducting backstage negotiations on the matter. We need to advertise this fact to the international community."