See this article, also from the Jewish Refugees blog: Yemenites Lived in Jerusalem since 1881.
In 1882, a group of Jews arrived from Yemen as a result of messianic fervor. Initially they were not accepted by the Jews of Jerusalem and lived in destitute conditions supported by the Christians of the Swedish-American colony, who called them Gadites. In 1884, the Yemenites moved into new stone houses at the south end of the Arab village, built for them by a Jewish charity called Ezrat Niddahim. This settlement was called Kfar Hashiloach or the Yemenite Village. Construction costs were kept low by using the Shiloach as a water source instead of digging cisterns. An early 20th century travel guide writes: In the “village of Silwan, east of Kidron … some of the fellah dwellings [are] old sepulchers hewn in the rocks. During late years a great extension of the village southward has sprung up, owing to the settlement here of a colony of poor Jews from Yemen, etc. many of whom have built homes on the steep hillside just above and east of Bir Eyyub,”
The Yemenite Jews living in Silwan were evacuated on advice of the British authorities in 1938, during the Arab revolt. After the 1948 Arab-Israeli War, Silwan was annexed by the Hashemite Kingdom of Jordan. It remained under Jordanian occupation until 1967, when Israel captured the Old City and surrounding region. Until then, the village had delegates in the Jerusalem City Council. (For the footnotes and sources, go to the original Wikipedia article).
To make my political position clear: it's true that there were Yemenite Jews living in Silwan until 1938, and the property they were living on was owned by Jews. It was, however, lost to Israel in the 1948 war, as was all of the Old City. At the same time, Arabs had to leave the Arab neighborhoods of what is now called West Jerusalem, most notably Baka and Katamon. The houses on the street I am currently living on in Baka appear to me to have mainly been built before 1948 (judging by the housing style). The house I am living is a renovated Arab house. If it is morally correct that the Yemenite Jews (or their descendants) should be permitted to return to Silwan, then why is it not equally morally correct for the Arabs of Baka and Katamon (or their descendants) to return and reclaim their houses? I oppose both returns, even though I think that there is a good moral case to be made in both cases, because we are living in the present, not in the past. In our present, the only possible solution, in my opinion, to the situation in Jerusalem, is to enable the city eventually to be the capital of both Palestine and Israel. In order for that to happen, there has to be a recognizably Arab part of the city which would be Palestine's capital. Jews resettling in Silwan or in Sheikh Jarrah are making that much more difficult, if not impossible.