Jonathan Scott and Zooey Deschanel were in it for the long haul from the very beginning.

During a recent tour of the couple’s Downtown Manhattan home with Architectural Digestthe New Girl star, 45, and HGTV host, 47, explained to the outlet that they first toured the property in late 2019 — only a few months after the pair first met while filming an episode of Carpool Karaoke.

“It was totally raw and stripped down, like an abandoned 1980s office building,” Scott said of their now maximalist duplex complete with French and English aesthetics.

Deschanel agreed: “It was definitely sketch.”

But the couple found charm and promise in the building, citing its prime location in the heart of New York City, high ceilings, large square footage and three terraces.

“You don’t really get a space like this in most New York apartments. We love that we have three private terraces. It just feels like we can get that little slice of what we have in L.A.,” Scott said, referring to the couple’s century-old residence in Los Angeles.

The couple, who got engaged in August 2023purchased their L.A. home in 2020. There, they live with Deschanel’s two kids: Elsie, 10 and Charlie, 8. That same year, the couple returned to New York to buy the two-level apartment as a secondary property. The couple partnered with New York-based interior designer Young Huh to bring their dream space to life.

Intricate and colorful wallpaper, custom millwork in the kitchen, a drop down projector screen, a vintage glass chandelier and even a 17th Century French tapestry are now featured throughout the home. Even Deschanel’s kids get their own space in Scott’s “favorite room in the whole house.”

“When the kids saw it for the first time, they like screamed,” Deschanels said. “They were so excited.”

Although the kids share a room while staying in New York, there’s separation between the beds in the form of more custom woodwork.

“I hope they know how lucky they are to have a contractor in the family,” Scott says of the kids.

The Elf actress shares her two kids with ex-husband Jacob Pechenik. The couple welcomed daughter Elise in July 2015 and son Charlie in May 2017. Although the couple split in 2019they still maintain a healthy co-parenting relationship.

Since the divorce, Scott has taken an active role in the kid’s lives as their “bonus dad,” though it hasn’t been easy.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“One thing that was a struggle early on was that the kids didn’t necessarily run up and jump in my arms and give me a big hug,” he explained in a June 2023 interview shared exclusively with PEOPLE. “I also remember several times where they’d be out with other kids and someone would say, ‘Oh your daddy’s so funny,’ and they’d be like, ‘That’s not my daddy, he’s my Jonathan.’ So that would hurt a little bit.”

He added: “In their mind, it’s not a mean thing to say — it’s how they know me.”