Kaguya, and the whole of Imperishable Night, is inspired by the Japanese legend of the Moon Princess, also known as the The Tale of the Bamboo Cutter, sometimes called the legend of "Kaguya-hime" (Princess Kaguya).
Her first name "輝夜" Kaguya, means "shining night", "sparkling night", or "radiant night". "蓬莱山" Houraisan refers to "Mount Hourai". And indeed, to not only be able to find the location and survive the impossible perils of ascending Mount Hourai (a legendary place filled with powerful youkai, mountain men, and other supernatural beings of every kind), but to also be able to enjoy a beautiful evening there, likely with some traditional Moon Viewing, would be to experience a truly otherworldly beauty: a shining night on Mount Hourai.
The last character "san" is based off the korean word san (산) or the chinese character shan (山)
Contrast Kaguya's name with "咲夜 十六夜" Sakuya Izayoi, a character speculated to have unrevealed connections to the moon and to Lunarians, whose name means "blossoming night" (Sakuya) and the "full moon on the sixteenth night of each lunar month" (Izayoi). They also both possess powers of time control.
Mount Hourai is the name of a legendary mountain that floats above the sea. Its inhabitants were said to possess, among other treasures, the elixir of immortality. It was also the destination of one of Kaguya-hime's impossible tasks.
In more serious doujin works, Kaguya is often depicted as having been driven insane or mentally unhinged as a consequence of her long life and inability to die permanently. However, in official works, namely Perfect Memento in Strict Sense, Inaba of the Moon and Inaba of the Earth and Cage in Lunatic Runagate, she is described as having a more cheerful and pleasant, if somewhat ditzy, personality, who likes to tease Reisen just like all the cast of Eientei. These different characterizations may stem from a more literal interpretation of the title of Kaguya's theme music, "Lunatic Princess"; believing it to be descriptive of Kaguya herself rather than the music.
It can be conjectured that Mokou and Kaguya's rivalry symbolizes for the cycle of the Sun and the Moon: Kaguya, the Lunar Princess, and Mokou, who uses as an in-game avatar a fire bird that strongly resembles a phoenix.
Kaguya was planned to debut as a playable character in Hisoutensoku, but was scrapped from the lineup due to time constraints. ZUN stated in an interview that her moveset as initially designed would have been overpowered and unbalanced the game and there was not enough time to properly balance her strength against the other characters (ironically the same issue was encountered with Mokou, who was also intended to be playable in the game).
Kaguya is one of the few characters that does not wear any type of headgear.
And also one of the few Touhou Characters to revive after being defeated by the heroines the first time in a danmaku fight.