RIYADH, Saudi Arabia – Saudi Arabia and Israel share a common enemy, Iran, and a common friend, the Trump administration in Washington.

But despite mounting evidence of informal cooperation, any open rapprochement—a goal of the Trump White House—between these two American allies remains elusive. That is largely because both have too little to gain, and too much to lose, from any such a breakthrough.

For the current Israeli government, the benefits of a diplomatic relationship with Saudi Arabia aren’t worth the tangible concessions to the Palestinians that Riyadh expects Israel to make in exchange. And to Riyadh, the price of being seen as forfeiting the Palestinian cause remains simply too high compared with what Israeli security assistance and technology, such as missile defense, could provide.

That has become especially so after President Donald Trump in December recognized the contested city of Jerusalem as Israel’s capital, galvanizing emotional protests across the Muslim world, reigniting support for the Palestinians—and prompting a rare rebuke from Riyadh.

Saudi Arabia’s ambitions to lead the entire Muslim world, or at a minimum a Sunni alliance opposed to Iran, are rooted in its control of Islam’s two holiest sites, in Mecca and Medina. As a standard-bearer of the faith, the Saudi kingdom can’t afford to be seen cozying up to Israel at a time when passions run high over the future of Jerusalem—home to Islam’s third holiest shrine, the Al Aqsa mosque.

The protests haven’t just been whipped up by Iran and its proxies, which seek to dismantle the Israeli state. Turkey’s President Recep Tayyip Erdogan, who seeks to challenge Saudi Arabia’s pre-eminence in the Muslim world, has been just as vocal. In this environment, any Saudi opening to Israel is guaranteed to be exploited by the kingdom’s rivals, and may even include a boycott of the hajj, or pilgrimage, to Mecca, a senior Saudi official cautioned.

“It’s a scary thought. Palestine is not an easy issue,” he said. “Saudi Arabia is expecting to hold Islamic leadership, and will not let it go easily. And, if you need Israel in anything, you can do it anyway, without having a relationship.”

Indeed, Israel and Saudi Arabia have already been cooperating discreetly by sharing intelligence and coordinating lobbying efforts and military activities that seek to deter Iran’s influence in the Red Sea.

Saudi Arabia is also making a public outreach to the American Jewish community. Senior cleric Mohammed al Issa, the kingdom’s former justice minister and the head of its proselytizing arm, the Muslim World League, in January sent an unprecedented official Saudi letter to the U.S. Holocaust Memorial Museum.

“We consider any denial of the Holocaust or minimizing of its effect a crime to distort history and an insult to the dignity of those innocent souls who have perished,” he wrote, offering a contrast to Iran’s sponsorship of Holocaust denial and revisionism.

Public contacts between Saudi Arabia and Israel, however, have been limited to informal meetings between retired officials at conferences. Saudi Arabia has balked at American proposals to allow overflight rights to Israeli civilian aircraft heading to Asia and, in December, refused to let Israeli chess players attend an international tournament in Riyadh.

The official Saudi position remains that Israel must accept the 2002 Arab League peace initiative, proposed by Saudi Arabia’s then-Crown Prince Abdullah, that calls for normalization of Arab states’ relations with Israel in exchange for Israeli withdrawal from Palestinian territories occupied in 1967, establishment of a Palestinian state with east Jerusalem as its capital, and a solution to the problem of Palestinian refugees.

That is a reality that Jared Kushner, Mr. Trump’s son-in-law and the White House point-man in attempts to reach a Middle East peace deal, acknowledged in remarks at a Brookings Institution event in December.

Countries such as Saudi Arabia “look at regional threats and they see that Israel, who was traditionally their foe, is a much more natural ally to them today than perhaps it were 20 years ago—because of Iran, because of ISIS, because of extremism,” Mr. Kushner said. “A lot of people want to see it put together, but we have to overcome that issue, the Israeli-Palestinian issue, in order for that to happen.”

With the right-wing government of Israeli Prime Minister Benjamin Netanyahu ruling out any compromise on Jerusalem and continuing to build settlements in the West Bank, Israeli acceptance of the 2002 Arab peace plan—or any other breakthrough in relations with the Palestinians—looks highly unlikely in the foreseeable future.

Even much more modest steps to address Palestinian grievances that could give Saudi Arabia cover for opening up to Israel are politically unpalatable for Mr. Netanyahu and his coalition partners, cautioned Joshua Teitelbaum, a professor specializing on Gulf affairs at the Begin-Sadat Center for Strategic Studies of Israel’s Bar-Ilan University.

“The price tag will have to be low.” Mr. Teitelbaum said. “Israel will not make any land concessions, or any concessions having to do with its security, for the sake of diplomatic relations with Saudi Arabia.”