The foggy wilderness of “Madeline’s Madeline” is difficult to peer through. Spellbinding shifts in cinematography from points of near clarity to ones of hazy confusion make the film’s visuals equally unintelligible. It is plain to see, however, that director Josephine Decker has not positioned ironclad boundaries around the execution and style of this feature. She gives herself more room to experiment — and experiment, she does.

At the center of this beguiling picture is Madeline (Helena Howard), a burgeoning 16-year-old who attends weekly rehearsals with a local theater group. Though she’s the youngest of the circle, her talent radiates, and she quickly becomes the subject of interest for the program’s ambitious director, Evangeline (Molly Parker). Outside her rehearsals, the film also examines the complicated dynamics of Madeline’s family. Subtle visual and audio cues indicate a private world to which only she, her brother and her mother, Regina (Miranda July), are privy. Something — though it’s never revealed what — isn’t quite right with Madeline’s mental health. There are hushed talks of missed pills and scenes of Regina’s frustration with Madeline’s vegan diet — or her inability to keep any food down. It’s through bewitching cinematography that Decker cocoons both the women’s exasperation with each other and Madeline’s declining mental health. The camerawork is the strongest element of the film, and it puts the audience in a headspace similar to Madeline’s when she’s around her mother.

The ceaseless instability of Madeline and Regina’s relationship often turns violent, reflected in the shocking cuts and incongruous focus of the cinematography. At one point, Regina berates Madeline after she kisses a boy, expressing her worries about Madeline having an episode and calling her unstable. On the word “unstable,” Madeline throws her drink in her mother’s face. At another, with explosive rage, Regina calls her daughter “Madeline the slut” after walking in on Madeline showing a collection of pornographic videos and pictures to her friends in her basement. Madeline rips a chunk of her mother’s hair out of her head in retribution. There are no emotions to read on Madeline’s face in either of these scenes, a testament to Howard’s superb creation of an equally threatening and withdrawn character. This is the Madeline Regina worries about — a removed and formidable teenage girl. This is also the Madeline Evangeline wants — a removed and formidable teenage actress.

As the movie picks up a bit of speed, the plot becomes murkier. The final sequence of the film lacks complete sense and lucidity as the theater troupe performs an ensemble for a frazzled Evangeline. It’s not clear why the characters break out into this outwardly spontaneous act, something it seems Madeline has organized. Perhaps it’s a rebellious turn against Evangeline’s dangerous exploitation of Madeline’s talent, or perhaps it’s a metaphor — though the latter would be a flimsy explanation for a scene filled with seemingly unconnected and ambiguous subject matter.

“What you are experiencing is just a metaphor,” one of the thespians says to the camera at the beginning of the movie. Whether this is spoken to the audience or to Madeline is not clear, and the following scenes provide no answer to how this is a metaphor. It leaves the viewer with a sense of disillusionment and a myriad of unanswered questions. Is the metaphor Madeline’s worsening mental stability? Is it a metaphor for the creative process? For artistic inspiration? For passion without limitation? And how many times can one see “Madeline’s Madeline” before the veil shrouding the metaphor can be cast away? It might be too many times to be worth it. The film is enchanting, but it too often gets caught in the whirlwind of artistic risk, and the effect can be nauseating.