You know the drill—when it comes to healthy eating, veggies should take up most of your plate, fiber keeps things moving through your system, and protein is a must for revving your metabolism.

Typically, a healthy diet consists of as many whole, non-processed foods as possible. That means plenty of non-starchy vegetables (like kale and eggplant), protein (found in eggs, chickens, and beans), whole grains (like brown rice), and healthy fats (found in olive oil, walnuts, and avocados), says Alexia Lewis, R.D.

According to the USDA and FDA, that also means limiting your intake of sodium, sugar, and saturated fats, which can all up your risk of heart disease and diabetes. (Limit saturated fat to less than 20 grams per day, sodium to less than 2,400 milligrams per day, and added sugars to less than 50 grams per day.)

But is it possible to take healthy eating too far? The experts agree: yes.

Thanks to many of the fad diets out there, we often get hung up on thinking of certain foods as ‘good’ and others as ‘bad,’ says Lewis. “We pick apart all the numbers and nutrients to make sure food is worthy of being eaten so we can believe that we are ‘being good,’” she says.

Problem 1: Eliminating Certain Healthy Foods

Fad diets often steer people toward cutting out certain food groups—with meat, dairy, and grains being some of the most common—in the name of ‘health.’ But unless you have a food allergy or intolerance, cutting entire food groups from your menu can backfire if you’re not careful.

Different foods provide different nutrients—and if you swear off those foods, you risk falling short on the nutrients they offer. Meat and dairy, for example, provide B vitamins, says Monya De, M.D., M.P.H., internist in Los Angeles. B12 (which is found in salmon, beef, milk, and eggs) is crucial for energy production and according to a study published in the Journal of Psychopharmacology, low levels have been linked with depression. If you cut meat and dairy from your diet, you’ll have to find other sources, like fortified almond milk. (The FDA advises we get six micrograms of vitamin B12 per day.)

Related: The 5 Key Nutrients You REALLY Don’t Want To Miss Out On

Dairy also provides calcium and vitamin D, which are hugely important for healthy bones. If you’re dairy-free, you’ll need to eat foods like broccoli rabe, oranges, and fortified almond milk for calcium, and fatty fish or fortified cereal for vitamin D, says Lewis. (The FDA recommends 1,000 milligrams of calcium and 400 IU of vitamin D a day.)

Problem 2: Overloading On Other Healthy Foods

On the flip side, our eating healthy quests can also lead us to load up on too much of certain foods and nutrients.

One of our favorites to overdo? Healthy fats. While monounsaturated and polyunsaturated fats offer heart health benefits, they’re super-high in calories and easy to overeat, says Lewis. One avocado, for instance, comes in at 240 calories and packs 21 grams of fat. Because one gram of fat packs nine calories, the FDA recommends sticking to just about 65 grams a day to prevent weight gain. Forgo proper portions when snacking on healthy fats, though, and you can easily surpass that recommended intake. So, limit the guac to about half an avocado-worth and follow serving sizes, says Lewis. That way you can reap the benefits of healthy fats without also expanding your waistline.

Protein is another that may be overeaten, especially by serious exercisers looking to reap its muscle-building, energy-boosting benefits. Since protein is harder to metabolize than other nutrients, eating more than you need can lead to constipation, says Daved Rosensweet, M.D., founder of iwonderdoctor.com. The backup is even more likely if loading up on protein also means you’re falling short on fiber. (We all have different protein needs; estimate yours here.)

Related: 7 Possible Reasons Why You Just Can’t Poop

Of course, you can overdo it on fibrous foods, too. Nosh on fiber-filled foods like black beans or broccoli all day long and you’re in for a serious case of the farts, since fiber can cause gas and bloating. Eating more than the recommended daily amount of fiber (25 grams a day for women and 38 grams a day for men) can help keep you feeling full and support weight loss, but eating as much fiber as humanly possible will not only leave you super-gassy, but may also reduce your absorption of other nutrients, like magnesium, calcium, and iron, says Lewis. Just concentrate on reaching your recommended daily amount of fiber and drink plenty of water to support easy digestion, she recommends.

Finding Balance

Eating healthy isn’t just physical—there’s a mental health component, too. When we strictly define what is and isn’t healthy—and when we try to stick to that 24/7—we set ourselves up for unnecessary guilt when we stray. We are only human, after all! A truly healthy diet is one that feeds both your body and mind, says Lewis. “One component of health is emotional health, and you should be able to enjoy treats without guilt or shame,” she explains.

Rest assured, The occasional splurge won’t impact your physical health, says Lewis. “One meal doesn’t make or break nutrition,” she says. “Even a week of unhealthy eating won’t have that much of an effect.” There’s no need to deny yourself that big bowl of pasta and side of crispy Italian bread every once in a while.

Eating a colorful, minimally-processed, nutrient-rich diet keeps your body well-nourished, but enjoying small treats nourishes your soul—the trick is to find your balance, she says.