The topic of male mental health has received much more prominence in the last decade – something many would argue is long overdue.

But while the high rates of male suicide have put the topic firmly on the national agenda – huge numbers of men still find it difficult to be openly vulnerable and discuss their feelings.

With suicide the biggest killer of men under 50 in the UK, a new film explores why many men still bottle up their emotions, and asks whether such ingrained, systemic behaviour can be changed.

Duncan Cowles, who directs Silent Men, is the first to admit his own struggles to open up emotionally to those closest to him, joking that tackling the subject for the documentary was “like dragging myself backwards through a hedge”.

But, he tells BBC News at the Sheffield Documentary Festival, he felt it was an important journey to go on because the risks of ignoring your emotions are far scarier.

“I think there’s still a lot of societal pressure around those traditional masculine traits,” Cowles says. “Being strong and things associated with being a leader or this stable, reliable, dependable figure.

“There’s maybe something still seductive about that idea of that old-fashioned man, that James Bond character that a lot of us maybe grew up with. I think it’s still quite sought-after by a lot of people.

“Whereas my experience is when you’re a bit more open and vulnerable, that can actually lead to stronger relationships, and therefore more joy, connection and fulfilment. And yet still we’re drawn to being closed off.”

• If you are affected by any of the issues in this story, the BBC Action Line has information about services that can provide support and advice.

The idea of men suffering in silence is nothing new. It was 1854 when Henry David Thoreau wrote: “Most men lead lives of quiet desperation.”

The fact that the essayist’s most famous observation has survived for 170 years and is still quoted so widely today is a testament to its relatability – but also of how little progress has been made.

Thoreau tapped into a feeling many men have, both then and now, of being unfulfilled – whether that’s in their career, relationships or for reasons they can’t quite put their finger on.

Overwhelmed by the commitments or mundanity of day-to-day life, while simultaneously unwilling or unable to voice their emotions, men often find it hard to cope.

“I suppose it’s always been something I’ve struggled with, opening up and showing emotions around family,” Cowles says of his reasons for making the film.

“While I was at university, I’d done a little film which investigated that, but it always felt like this thing that I hadn’t fully resolved in my life.”

Setting out to explore the subject further, Cowles posts an advert online, looking for men to speak to for the documentary. He then goes to interview some of those who reply.

He travels around the country and speaks to men from different walks of life over several years. At one point, he attends a retreat where men gather for a weekend to do group therapy.

“I wanted to go on my own journey, parallel to speaking to men as part of the film. But it turned into a more personal film than I expected,” he explains.

The Scottish filmmaker interviews members of his own family and studies how their personalities shaped his own. “My dad likes drilling holes in the wall, and his stamps, but he’s not big on verbal communication,” Cowles notes – painting a picture of fathers many will recognise.

One trauma specialist interviewed for the film cites a study which found that babies in blue blankets – and therefore assumed to be boys – were touched for significantly less time than girls.

“So that’s really interesting, what are we doing in our culture about socialising boys to not be touched as much, not be seen as vulnerable, not need emotional support as much as girls?” she asks.

“The toughening up – ‘be a brave boy, big boys don’t cry’ – that process of making you into a man, where someone has told you that that’s what a man is, that a man has got to be emotionally strong.

“There are all these social constructs that kind of trap you, which is causing huge numbers of mental health issues for guys. We’ve got a big problem on our hands.”

Silent Men is Cowles’ first feature documentary, but he previously won a Bafta Scotland award for his short film Isabella.

Asked if it was difficult to get the men featured in the film to talk about their feelings, Cowles replies: “Strangely not, because there’s something about speaking to a stranger that can sometimes feel a lot easier than speaking to someone who’s in your own family. I guess it’s because you’ve less to lose.”

The insights from other men make for interesting viewing. One contributor, Ainslie, says his outlook changed after having a baby. But not necessarily in the way you might think – he says the responsibility of being a father actually meant there was less time and opportunity to find room for expressing his own emotions.

Another man, Dom, tells his young son constantly how much he loves him, and says he hated the thought of anybody close to him dying without knowing how much they meant to him.

One of the film’s most affecting – and funniest – moments comes when the microphone Cowles’ brother is using cuts out after the battery dies, and he uses that exact moment, when he can’t fully be heard, to tell Duncan: “I love you.”

Despite all the heavy talk, the film is peppered with moments of levity – the director occasionally cuts away abruptly from tough conversations to run footage of soothing shorelines and bumble bees on flowers.

“One thing I really wanted to do with this film is make it feel accessible, not all mental health based, but [to] also contain humour and leave people with a smile on their face,” he says.

The documentary builds towards a conclusion as Cowles sits down with his parents to tell them he loves them.

While both are receptive, his father is somewhat dismissive of the “airy-fairy” nature of showing emotions, suggesting it’s “not a normal thing for a man”.

Therefore, his father argues, if somebody isn’t opening up, that’s actually a good sign, as not feeling the need to be vulnerable means they are generally feeling content.

Reflecting on his father’s comments now, Cowles says: “He comes from a generation where he doesn’t want to see someone visibly upset, so if you’re showing nothing, then that’s a good thing [to him] because it means you’re OK. I know what he means.

“Younger people seem to have a different opinion on these kinds of things, and they are more willing to open up,” he notes. “There’s that awareness of the issue.”

Cowles admits on camera his worry that he will pretend to have made progress for the purposes of giving the film a happy ending, when in reality there may not have been any significant change.

But several months after filming wrapped, Cowles says making the film “definitely has” altered his behaviour.

“I wouldn’t say it’s a Hollywood transformation,” he laughs. “It’s been quite a gradual thing, and I wouldn’t say I’m perfect by any means.

“But compared to what I was like before, my partner is very appreciative that I’ve changed, that I’m more conscious of it and able to recognise when I should open up a bit more.”

Silent Men is screened at Sheffield DocFest on Friday and released more widely later this year.