The Magic Castle has been a Los Angeles landmark for more than 40 years, but club founder Milt Larsen says the only trick to its longevity is the simple appeal of magic itself.

“There seems to be no end to the popularity of magic,” says Larsen, 75. “There’s no way you can watch somebody doing a coin trick or a card trick and think about anything else.”

Larsen’s life has been all about magic, a business he and his late brother, Bill Larsen Jr., were literally born into. His father, Bill Sr., gave up a career as a defense attorney to perform in a touring magic show with his wife, Geraldine. In 1936, he founded Genii, a magazine for magicians that’s still published today. When World War II made touring difficult, the Larsens settled in Pasadena and ran a magic shop in addition to the magazine.

“I got to meet some of the legendary people in magic because they were over at the house,” says Larsen.

Comedy was another of Larsen’s passions. He put out some joke books with a friend while still in high school and sold them through ads in Billboard and Variety.

“Nobody knew we were kids,” he says. That made the TV business a natural for Larsen — who calls Walt Disney his role model — and his brother. Milt wrote for “Truth or Consequences” for 18 years while Bill Jr. worked his way up to an executive job at CBS.

But Larsen had not forgotten magic. Milt and Bill put on their first “It’s Magic!” stage show in 1956. And it was from his office, at the present site of Hollywood & Highland, that Milt spotted a three-story Victorian in the Hollywood Hills. “It looked like a haunted house and in a little bit of disrepair at the time,” Larsen says.

It was an ideal spot for the magician’s clubhouse Larsen’s father had spoken of before his 1955 death.

Milt worked out a deal with the house’s owner, Tom Glover, and renovated it himself. Bill Jr. came in to run the business side of things and the castle opened on Jan. 2, 1963.

“At the time, it was just a small bar and a room where a magician could do close-up acts,” Larsen says. “We were the first ones to create a stage for close-up magic. Before that, there were really no close-up venues where a magician could go in and have lights and go table-hopping.”

The Academy of Magical Arts & Sciences became active when the castle opened its doors. Bill Jr. developed it into an organization that promoted magic and magicians, with membership now topping 5,000.

The club has always been open only to members and their guests. That created a mystique that was aided by some of the more famous members.

“It wasn’t unusual to find Cary Grant, Orson Welles and Johnny Carson hanging out at the castle,” Larsen says.

Larsen continued to do some work in Hollywood, consulting on 1971 animated Disney film “Bedknobs and Broomsticks” and acting in a 1981 episode of “Hart to Hart.”

While the castle has been around for more than 40 years and the “It’s Magic!” show for 50, Larsen says magic keeps everyone young.

“There’s no such thing as an old magician, because we never grow up.”