People: Simon Cowell Runs the Show at AGT and Has Rubbed a Lot of People the Wrong Way

Simon Cowell's influence behind the scenes of "America’s Got Talent" is starting to rub many of the show's employees the wrong way.

It is no secret that Simon Cowell has significant influence on "America’s Got Talent." However, a source tells PEOPLE his role is becoming problematic, it is revealed:

"It's true that America’s Got Talent is a boy's club. Simon runs the show and as long as he's there, he will continue to do so. Everything goes by him and he's rubbed a lot of people the wrong way."

As a judge and executive producer, the source says he practically "runs the show" and when it comes to getting his way, people rarely say no to him.

The comments about Cowell come days after the controversial firing of judges, Gabrielle Union and Julianne Hough.

Reports have surfaced that Union was fired for speaking out against racial insensitivities on set and complaints made by staff workers to her that her hairstyles were too black for the "AGT" audience.

Former judge on the show, Howard Stern believes Cowell had a role to play in Union's firing. He says Cowell orchestrated her firing to keep the men on the show and to keep the women in rotation.

Stern who was on the show for four seasons alleges that Cowell has set up something like a boy's club, he explains:

"What he manages to do on all his shows is he constantly replace the hot chicks with hotter chicks and younger chicks, which is so obvious. Simon (and) Howie get to stay, but the three women had to go: Tyra Banks, Mel B., Heidi Klum, and they find two younger, hotter women."

Union has not released any official statements about the controversy surrounding her firing. But she has posted cryptic messages on her social media alluding to her going through a difficult time.

Julianne, who was also fired along with Union, allegedly dealt with staff complaints about her wardrobe and hairstyles. However, in a statement to Variety, she denies having a bad experience on the AGT set.

On Sunday, NBC and Syco Entertainment, which produces AGT, released a statement and expressed their commitment in ensuring that all their employees enjoy a respectful workplace culture.

The statement mentions that they are working with Union, through her representatives to learn more about her situation and will take corrective action.