Wilford Woodruff kept a journal. In one entry, he wrote that Brigham Young said that Joseph Smith returned the golden plates to a cave in the Hill Cumorah. So this is not a primary source (from Joseph himself) but interesting:
December 11, 1869: At School of the Prophets: President Young said in relation to Joseph Smith returning the plates of the Book of Mormon, that he did not return them to the box from where he had received them. But he went into a cave in the Hill Cumorah with Oliver Cowdery and deposited those plates upon a table or shelf and in that room were deposited a large amount of gold [170] plates, containing sacred records; and when they first visited that room, the sword of Laban was hanging upon the wall and when they last visited it, the sword was drawn from the scabbard and lain upon the table, and a messenger who was the keeper of the room informed them that that sword would never be returned to its scabbard until the Kingdom of God was established upon the earth and until it reigned triumphant over everyone. Joseph Smith said that cave contained tons of choice treasures and records. (Wilford Woodruff Journal)
Another source of this story is when Brigham Young was in Farmington, Utah, organizing a stake on June 17, 1877. Brigham shared the story from the life of Oliver Cowdery — when Oliver and Joseph went to the Hill Cumorah to return the plates, and that it opened into a cave:
I believe I will take the liberty to tell you of another circumstance that will be as marvelous as anything can be. This is an incident in the life of Oliver Cowdery, but he did not take the liberty of telling such things in meeting as I take. I tell these things to you, and I have a motive for doing so. I want to carry them to the ears of my brethren and sisters, and to the children also, that they may grow to an understanding of some things that seem to be entirely hidden from the human family. Oliver Cowdery went with the Prophet Joseph when he deposited these plates. Joseph did not translate all of the plates; there was a portion of them sealed, which you can learn from the Book of Doctrine and Covenants. When Joseph got the plates, the angel instructed him to carry them back to the hill Cumorah, which he did. Oliver says that when Joseph and Oliver went there, the hill opened, and they walked into a cave, in which there was a large and spacious room. He says he did not think, at the time, whether they had the light of the sun or artificial light; but that it was just as light as day. They laid the plates on a table; it was a large table that stood in the room. Under this table there was a pile of plates as much as two feet high, and there were altogether in this room more plates than probably many wagon loads; they were piled up in the corners and along the walls. The first time they went there the sword of Laban hung upon the wall; but when they went again it had been taken down and laid upon the table across the gold plates; it was unsheathed, and on it was written these words: “This sword will never be sheathed again until the kingdoms of this world become the kingdom of our God and his Christ.” I tell you this as coming not only from Oliver Cowdery, but others who were familiar with it, and who understood it just as well as we understand coming to this meeting, enjoying the day, and by and by we separate and go away, forgetting most of what is said, but remembering some things. So is it with other circumstances in life. I relate this to you, and I want you to understand it. I take this liberty of referring to those things so that they will not be forgotten and lost. (Journal of Discourses 19:38.)
Original 2017, Updated 2021
