One may conventionally think of fiction as stories authors have invented -- out of the ether, as it were. But at times, not just inspiration, but certain details, come from life, viewed or experienced.
The first character one encounters in "Ulysses," -- 'stately, plump Buck Mulligan' -- is largely based on a man named Oliver St. John Gogarty, and to those who knew him, obviously so. Likewise, various other characters in James Joyce's epic can be linked to actual people. And Joyce clearly incorporated versions things that happened to him in his book.
In my novella, "Manhattan Morning," the final scene at an eatery in Grand Central Terminal is very close to an actual occurrence. After I had the experience, I thought it would work well as the final scene of a story, serving as a foil, mainly in the realm of values, to what had gone before. It is also a sympathetic and somewhat poignant portrait of a woman attempting to balance a demanding job and motherhood in a hectic world -- a good story.
I doubt that readers find such linkages distressing. But what about the reverse -- when something represented as fact turns out to be made up?