There are many actions we can take in a tabletop role-playing game, both as players and as Dungeon Masters. However, many will be sub-optimal. Such options aren't a bad thing, since they will often make sense due to the characters involved or situational reasons. They also help add character to a game, since there wouldn't be much of a point if players always just did the optimal thing in every situation. However, managing and account for them can be a bit of a challenge. With that in mind, I hope to share some of my thoughts on the matter.
Players Will Naturally Do So
When you design a combat encounter, in particular a challenging one, there will often be things we include that can be exploited to make things easier. Even more generally, some creatures have strengths and weaknesses which also account for their difficulty. However, we can't reasonably expect a player to just instantly know the weakness of the creature. In some cases they will because they have experience, but they might still act otherwise since it makes more sense for their character.
Dungeon Masters Often Optimize
I know that when I'm doing a combat encounter, I can optimize when I don't really want to. When fighting an intelligent creature there should definitely be a feeling of intelligence in the opponent. However, intelligent doesn't mean omnipotent. It can be immersion breaking if the opponent seems to perfect. Even worse, it can force your players to optimize as well. While combat is also about choosing a good option, it's also a way for a player to bring their character to life. The more optimized our Dungeon Master plan, the more players tend to optimize as well since players typically don't want their characters to die. They aren't against an honest loss or death, but they will try to keep their characters alive. This also includes meta-gaming. I find it's far more likely to have players use their out-of-game knowledge when they know they'll need it to have a fighting chance.
Account In Difficulty
I think I've made it known that my opinion on encounter difficulty is that the rules provided are guidelines. They can be a good place to start, but the situation will also greatly contribute to the difficulty or ease your players encounter. The more intricate the encounter, the longer it might take to find a solution. Again, I say a solution because there should be multiple. Otherwise it gets too rail-road-y and too optimized, running into the issues from the previous section. What it means in practice is that there may need to be a turn or two in order to let the players fumble for an answer.
How long of a buffer we should give or how to adjust difficulty is not an easy question and again, is very situation dependent. It also depends how clear things are. Having part of the encountered enemy fall back to a better position allows the players to take out some of them isolated, making the encounter easier. However, they then have to solve the issue of the superior position. They may choose to wait for the enemy to leave or attack. They could also spend a couple turns scouting and coming up with a solution. Weaknesses, such as areas of sunlight (particularly good against vampires or shadows), particular kinds of damage (anyone who has fought a troll will know this one), may also take some time to effectively exploit. If we also want to have players take sub-optimal actions because their character wouldn't know about the weakness, it might make sense to send a couple fewer enemies or at the very least have them act in such a way that they have a chance. Part of it is trust. If the players trust that things are still do-able even if they don't optimize as much as possible, they'll for more comfortable with improvising and fumbling for a solution. Turns tend to go faster too, since they feel more comfortable acting instead of deliberating as a team .You can of course try to force them to act quicker, but combined with everything else this can lead to resentment.
Optimization Feedback Loop
I briefly touched on it, but these kinds of situations can form a bit of a feedback loop. You'll optimize all your encounters, which forces the players to optimize theirs, which forces you to optimize again in order to challenge them, and so on. As players get more accustomed to the rules you'll see this as well. This case is to be expected. However, it's important not to go too far and take away player choice and expression. The problem is in the freedom. It constrains things further because sub-optimal actions are always to be avoided in this case. Anything else is probably death. There is, of course, always some wiggle room too because of the element of chance. However, that can go in either way.
We can also reward sub-optimal actions. If we are playing D&D 5th edition, we might give inspiration for them every now and then if it still is inline with the character. Another one I've seen is extra experience. The problem here is that it can shift things too far the other way because the external rewards make it more advantageous to take the previously sub-optimal action. If required this can be solved with keeping the danger real, forcing a choice between the rewards and staying alive. Not consistently rewarding the behaviour may also be enough. These considerations are why I personally find this method of encouraging sub-optimal actions to be difficult to employ, though I've seen them work well as a player before.
Clear Optimal Actions
A big issue here is that an encounter or situation has an optimal action. I designed it like that, but I really didn't want to. It just happened. What I find is it often helps to either design the encounter with multiple in mind, or do a quick check at the end that there are at least a few. If we can quickly think of a bunch, most often there are many more that are there as well from my experience.This is important since it will still allow a situation to be fun even if optimization occurred on both sides. This is because it's still a problem to be solved instead of a solution to be applied. Having an encounter that is nothing but applying a solution isn't fun. Even if they have some item that should be exploited, getting themselves in a situation to do so will require the players to solve a problem first. It also might mean that a sub-optimal action this turn may lead to a better optimal action next turn, forcing a trade-off. An obviously easy encounter may also provide situations for players to express themselves, while also saving resources for later.