We all know our hormones are important for our health—but how much further does your knowledge on the subject extend?

Thought so.

Hormones are chemicals that act like messengers, traveling through our blood to control various body functions, like our blood pressure and heart rate, bathroom habits, hair growth, libido, and sleep, according to the U.S. National Library of Medicine.

There are many types of hormones, including sex hormones (estrogen, progesterone, and testosterone), thyroid hormones, cortisol, insulin, glucagon, and many more, explains Allison Betof Warner, M.D., Ph.D., medical oncology fellow at Memorial Sloan Kettering Cancer Center. All of these hormones have different jobs—insulin helps transport sugar to our cells for energy, for example—but they work in tandem to keep our bodies functioning. So when one hormone is off, it affects the whole system, creating what we know as a ‘hormonal imbalance,’ says Warner.

When these chemicals get thrown out of whack, they can cause absolute chaos in our bodies. Often, hormonal dysfunction leads to fatigue, weight gain or loss, headaches, mood fluctuations, acne, insomnia, or digestive problems. Doc-approved plans of action for balancing out different hormonal issues will vary—but chances are they’ll all look at your food habits. Why? Because it turns out the food you eat can impact your hormones more than you think.

Check out the six foods experts say could potentially contribute to hormonal woes:

1. Coffee

Most of us drink coffee precisely because of how it affects our bodies. That jolt of energy is often so needed in the morning, and then again by mid-afternoon.

The caffeine in coffee causes the body to boost its production of a hormone called cortisol, explains holistic nutritionist Kelly LeVeque. A stress hormone, cortisol helps all of our cells communicate. Normally, our cortisol levels are highest in the morning to wake us up, and lowest before bed so we can wind down and fall asleep, according to research published in Therapeutic Advances in Endocrinology and Metabolism. (The natural fluctuating pattern of cortisol is a part of our ‘circadian rhythm.’) Maintaining normal cortisol levels and a circadian rhythm helps our immune system function properly—and when our cortisol goes haywire, we may end up with sleep problems, poor blood sugar regulation, a slower metabolism, weight fluctuations, decreased immunity, and potentially even feelings of anxiety or depression, explains LeVeque.

Hence why coffee—no matter what time of day we’re drinking it—can leave us feeling jittery or nervous and make it difficult to fall asleep at bedtime. To avoid jacking up your cortisol levels, LeVeque recommends limiting coffee intake to one cup in the morning, and having a gentler cup of tea in the afternoon instead of another round of joe.

Related: 5 Natural Sources Of Caffeine—Other Than Coffee

2. Sugar

The undeniable truth: Our bodies need some sugar. Our cells store the sugar (a.k.a. ‘glucose’) we consume as glycogen, which we can use for energy later. When we eat sugar, the hormone insulin helps transport that sugar out of our blood and into our cells.

When we eat too much sugar and our body churns out tons of insulin, our cells eventually become resistant to it (a condition called ‘insulin resistance’) and extra glucose is left in our blood stream. This extra glucose is stored as fat, leading to weight gain and putting us at risk for type 2 diabetes, according to the CDC. So it’s no surprise that research published in JAMA found that women who drank one sweetened beverage or more a day had up to an 83 percent higher risk of developing type 2 diabetes over the course of a decade.

People with a family history of insulin resistance or diabetes are more likely to deal with the negative side effects associated with going overboard on sugar, says Warner. However, none of us are immune to the hormone-altering effects of consistent late-night candy bars, or a diet high in Skittles, so Warner recommends only consuming sugar in moderation. (The CDC suggests we limit added sugar to 10 percent—or less—of our daily calories. That’s about 50 grams or 12 teaspoons for someone following a 2,000-calorie diet.)

Related: The 5 Fruits With The Most (And Least!) Sugar

3. Carbohydrates

When we think ‘sugar,’ we think about stuff we just talked about, like Halloween candy and pints of ice cream. Here’s the thing, though: Since our bodies break carbs down into glucose, they can potentially have the same effect on our hormones as the straight-up sweet stuff, explains Elena A. Christofides, M.D., F.A.C.E. The main culprit here: refined, processed carbs. (Think white bread, muffins, and pasta.)

In addition to possibly screwing with our insulin, carbs can also affect other hormones. When we eat carbs, our bodies release the feel-good hormone serotonin—hence why that Sunday morning bagel tastes so dang good. But this carb-serotonin connection also explains why we crave carbs when we’re stressed out (and likely have high cortisol levels), says Christofides. And that’s a recipe for a major mood and energy rollercoaster.

Instead of going cold turkey on carbs (because what kind of life is that…), think about carbs in terms of their glycemic index, Christofides says. Glycemic index is a measure of how much a food will spike your blood sugar; the lower the number—and resulting blood sugar spike—the better for your hormones, she explains.

Basing your carb intake in complex carbs is your best bet at keeping those hormones stable. Foods like apples, lentils, and beans are all carb sources with low glycemic loads—and that also provide fiber and vitamins our bodies need.

4. Meat

We’d be lying if we didn’t admit our love for a good burger—but about the consistent debate over whether meat (namely non-organic or hormone-treated meat) is an enemy to our hormonal health definitely makes us think twice.

A quick recap on the drama: In 1999, the European Union’s Scientific Committee for Veterinary Measures Relating to Public Health proposed that the seven hormones commonly found in treated meat products (testosterone propionate, trenbolone acetate, estradiol, zeranol, progesterone, melengestrol acetate, and bovine somatotropin) could potentially threaten our health in a number of ways. It had been partially banning imports of hormone-treated meats since 1981, according to the Congressional Research Office.

Some research suggests these hormones have adverse effects on the health of animals, but large-scale studies have not concretely identified a negative impact of these hormones on humans, according to a risk assessment paper published in Toxicology Research. “We don’t have any way to confirm or deny that hormones-treated meats affect us yet” says Christofides—which is why experts don’t always agree on how (or if) to incorporate meat into your diet.

Meanwhile, the FDA has maintained that because these hormones are found in such small amounts in meat, they don’t pose any threats to human health. (The FDA has approved of the administration of hormones to livestock to help them grow faster since the 1950s.)

While LeVeque recommends decreasing meat consumption overall, Christofides suggests choosing organic meat, eggs, and dairy products whenever possible. As researchers continue to look into the long-term effects of hormone-treated animal products, it’s better to be safe than sorry, she says.

Keep in mind, though, if you are a vegetarian (or are considering going meatless), that plant proteins and animal proteins aren’t exactly equal. Animal proteins are considered ‘complete proteins’ and contain the essential amino acids our bodies can’t produce on their own, but plant proteins are not complete proteins and do not contain adequate amounts of all of these amino acids.(Amino acids are molecules that make up protein that our body needs for a number of functions.) Plus, meat also provides B vitamins and iron, two important nutrients that are harder to find in plant sources.

The less meat there is in your life, the more important it is to regularly include complete proteins like eggs and dairy. Make sure you’re eating a variety of plant-based protein sources, too, or consider a protein supplement.

The bottom line is that there is no bottom line. If you’re a carnivore but concerned about hormones, go for organic, grass-fed meats. And if you have any existing hormonal issues, talk to your doc or a dietitian about how to tailor your diet for your healthiest self.

Related: 7 Protein Sources for Vegetarians

5. Soy

Chances are, you’ve heard quite a bit of back-and-forth about soy. Much of the debate is over phytoestrogens, which are plant compounds that act similarly to the hormone estrogen, according to research published in Front Neuroendocrinology. These phytoestrogens are often used to replace some of the estrogen women lose during menopause, and are found in some other foods. Soy and foods made from it (like tofu and tempeh), though, are the most common food source of phytoestrogens, says LeVeque.

According to LeVeque, phytoestrogens can potentially prevent actual estrogen from binding to its receptors, and increase how much estrogen is then floating around in your blood. (One small study published in Cancer Biomarkers, Epidemiology and Prevention, for example, found that about 30 percent women who ingested 38 grams of a soy protein supplement regularly had increased levels of estradiol—a.k.a. estrogen—in their blood after three months.) This may, in turn, influence testosterone and thyroid hormone levels, LeVeque says. (Remember, when one hormone is whacked out, it can affect the levels of others.)

Overall, research on whether these phytoestrogens are detrimental to or beneficial for our health is mixed. Studies have found myriad results. According to a review published in German Medical Science, they’ve shown both positive and negative correlations between phytoestrogen consumption and certain cancers in women, demonstrated that phytoestrogens support women’s health post-menopause, and also suggested that phytoestrogens can negatively impact fertility. A conclusive, across-the-board verdict, though, just doesn’t exist.

What the research does suggest is that your age, sex, and health status determine how phytoestrogens may or may not affect your body. For most people, noshing on soy occasionally is no big deal, says Christofides. However, since estrogen-blocking therapy is often used in the treatment of women with sex-related cancers, consuming phytoestrogens under these circumstances is a different ballgame. “It could literally affect how effective treatment is,” she says.

She also suggests kids, whose hormones are changing and sensitive, should avoid taking in extra phytoestrogens. (While some animal studies suggest phytoestrogen intake can influence sexual development, conclusive human research is lacking, according to a review published in the Korean Journal of Pediatrics.)

6. Alcohol

You’ve heard it before: Alcohol opens up a Pandora’s box of chaos in your body—and that includes messing with your hormone function.

According to the National Institutes of Health (NIH), alcohol not only affects our body’s production of blood sugar-controlling insulin, but can also impact our sexual hormones, potentially knocking down testosterone in men and derailing women’s menstrual cycle.

Related: 11 Ways You’re Sabotaging Your Metabolism

In addition to its impact on our metabolic and reproductive function, alcohol can also throw off our stress hormones. At first, alcohol makes us release a rush of the feel-good hormone serotonin, says Christofides. But then, when we’ve used up our serotonin, we’re left feeling pretty down.

Friendly reminder: The USDA defines ‘moderate drinking’ as up to one drink per day for women and up to two drinks per day for men. When it comes to alcohol and your hormones (and health), less is more.