Kaguya, and the whole of Imperishable Night, is inspired by the Japanese legend of the Moon Princess, also known as the The Tale of the Bamboo Cutter, sometimes called the legend of "Kaguya-hime" (Princess Kaguya).
Her first name "輝夜" Kaguya, means "shining night", "sparkling night", or "radiant night". "蓬莱山" Houraisan refers to "Mount Hourai". And indeed, to not only be able to find the location and survive the impossible perils of ascending Mount Hourai (a legendary place filled with powerful youkai, mountain men, and other supernatural beings of every kind), but to also be able to enjoy a beautiful evening there, likely with some traditional Moon Viewing, would be to experience a truly otherworldly beauty: a shining night on Mount Hourai.
The last character "san" is based off the korean word san (산) or the chinese character shan (山)
Contrast Kaguya's name with "咲夜 十六夜" Sakuya Izayoi, a character speculated to have unrevealed connections to the moon and to Lunarians, whose name means "blossoming night" (Sakuya) and the "full moon on the sixteenth night of each lunar month" (Izayoi). They also both possess powers of time control.
Mount Hourai is the name of a legendary mountain that floats above the sea. Its inhabitants were said to possess, among other treasures, the elixir of immortality. It was also the destination of one of Kaguya-hime's impossible tasks.
A silhouette of Kaguya occupies the front of Imperishable Night's jewel case.
Kaguya and Mokou fight to the death every chance they get. Their relationship has been characterized as a constant "Tom and Jerry" back and forth battle.
However, in Inaba of the Moon and Inaba of the Earth, Kaguya and Mokou's relationship seems much more casual instead—though that might be due to the carefree attitude of the author...
In more serious doujin works, Kaguya is often depicted as having been driven insane or mentally unhinged as a consequence of her long life and inability to die permanently. However, in official works, namely Perfect Memento in Strict Sense, Inaba of the Moon and Inaba of the Earth and Cage in Lunatic Runagate, she is described as having a more cheerful and pleasant, if somewhat ditzy, personality, who likes to tease Reisen just like all the cast of Eientei. These different characterizations may stem from a more literal interpretation of the title of Kaguya's theme music, "Lunatic Princess"; believing it to be descriptive of Kaguya herself rather than the music.
Kaguya is also the name of a Japanese lunar-orbiting probe launched in September 2007 as part of Project SELENE to map the moon's surface. At the time of the launch, fanart depicting Kaguya being "sent back to the moon" aboard the satellite, usually against her will, was popular (one tribute video even superimposed her theme music over footage of the launch of the rocket). The same probe was later intentionally sent crashing into the moon, giving way to a new surge of fanart. (Example)
It can be conjectured that Mokou and Kaguya's rivalry symbolizes for the cycle of the Sun and the Moon: Kaguya, the Lunar Princess, and Mokou, who uses as an in-game avatar a fire bird that strongly resembles a phoenix.
Kaguya was planned to debut as a playable character in Hisoutensoku, but was scrapped from the lineup due to time constraints. ZUN stated in an interview that her moveset as initially designed would have been overpowered and unbalanced the game and there was not enough time to properly balance her strength against the other characters (ironically the same issue was encountered with Mokou, who was also intended to be playable in the game).
Kaguya is one of the few characters that does not wear any type of headgear.
And also one of the few Touhou Characters to revive after being defeated by the heroines the first time in a danmaku fight.