Let's imagine you created a new feature and provided a feature flag for it. Required tests were added for "on" and "off" states, but one assertion was left without update for "on" state. You pushed your changes, the flag is "off", all tests are green. Yay! Changes are merged, code deployed, the feature flag is toggled and feature is released.
Once the feature flag is toggled the very next CI run is failing. Since we forgot to update assertions for "on" state we troubled others and have to take care immediately.
That can happen if you run e2e tests, use dynamic feature flags that can be toggled in a runtime and same set of flags used for staging enviroment. I have experience using split.io and it's a great tool, but dynamic power brings new challenges like I described above. What we can do about it?
Use less feature flags
Ease of adding a feature flag to every change comes with burden of mantaining, code and test complexities, forgotten flags.
Use static configuration
Yes tools like Split.io brings friendly UI that can be used by anyone in the company anytime, but most if you have efficient CI\CD in place toggling via configuration change sounds reasonable. You'll make sure that application will be tested against configuration it's supposed to run with.
Run tests against both states before release
Before releasing do extra run with "on" state to spot potential broken tests. Yes, easy to forget and something to do manually, but generally speaking, might help.
Don't use production set of flags on stages
"Toggles on production enviroment shouldn't inpact staging enviroment" you might say. "But who then we are going to test it?" might someone argue.
The list goes on and on and I don't have a setteled answer for your team apart from the first two points mentioned. Would like to hear your experience and don't forget to delete an obsolete feature flag ;)
May 12, 2020