Apache mod_fcgid, Missing Environment Variables, and INITENV_CNT
Apache mod_fcgid has hard coded limits for environment variables.
I learned this the hard way: Failures in production that weren't observed in development or testing. After a deployment, environment variables were randomly missing from mod_fcgid apps in production.
This was especially confusing to troubleshoot, because each fcgid process was missing different variables. For example, one thread generates SMS messages just fine, while another one throws errors about missing an api key. Different processes had dropped different environment variables!
Environment Variables with mod_fcgid
Apache protects your possibly sensitive operating system environment variables from your fcgid servers. A perfectly sensible design decision. By default, the application receives an empty environment.
The FcgidInitialEnv directive is used in an Apache configuration to set environment variables available to the fcgid server.
# httpd.conf
#
# Simple deployment of a perl Catalyst application under mod_fcgid
Alias / /opt/howdy-world/script/howdy_world_fastcgi.pl/
FcgidInitialEnv GOOGLE_API_KEY abcd-12345-efgh-67899
FcgidInitialEnv MAILGUN_ENABLED 1
FcgidInitialEnv MAILGUN_API_KEY ijkl-0987-mnop-6543
FcgidInitialEnv MAILGUN_DOMAIN mg.example.com
FcgidInitialEnv MAILGUN_SENDER Howdy <noreply@example.com>
<Directory /opt/howdy-world/script>
SetHandler fcgid-script
Options +ExecCGI
AllowOverride All
Require all granted
</Directory>Unexpected Hard Coded Limits
The configuration reference for FcgidInitialEnv neglects to mention the three hard coded limits of this feature. Reasonable. Many generations ago, when the code was written, you might be considered ridiculous to believe they were too low. After all, 640k is enough ram for anybody.
Here's the limits:
Maximum number of environment variables: 64, or 1024 in RHEL
The original source tree has this set as 64. Redhat increases this limit to 1024 at compile time from RHEL 8.4 according to their bugtracker.
Maximum size of environment variable key: 64 characters
I'm not sure what would happen if you attempted to exceed this limit. I presume your environment variable would be unavailable to your server.
Maximum size of environment variable value: 128 or 256 characters
On the trunk branch, this limit was updated from 128 to 256 characters in 2018 with commit 03b9832a. This change is not reflected in the changelog, from what I can tell. I didn't delve deep enough to know if the change has upstreamed, and if so when and what version. If somebody cares enough to figure that out, email me and I'll update this section.
Apache Error Log Message
[Wed Mar 30 12:52:50.188571 2022] [fcgid:warn] [pid 3844:tid 140189] [client 1.2.3.4:63534] mod_fcgid: 6 environment variables dropped; increase INITENV_CNT in fcgid_pm.h from 64 to at least 70, referer: https://howdy-world.example.com/
This log message is easy to miss, because it may not appear for every vhost when they all have their own log files. Or maybe it does, and I'm blind. I haven't circled back to confirm this. But I believe this message only appeared once, in the logs for the alphabetically first vhost using mod_fcgid.
Possible Solutions When Affected By These Limits
Reduce reliance on environment variables
This is what I'm going to do. The Perl Catalyst applications I have in production provide integrated config file tooling. I've been using environment variables a little excessively, in place of separate development/staging/testing/production application config files. Much of what I've configured with environment variables could happen elsewhere. I'll keep environment variables only for secrets and failsafe feature flag kill switches.
Switch to a linux distro that increases these limits at compile time
Red Hat has helpfully done this. Not aware of others, but maybe.
Compile Apache from source in production
If you're not already doing this, for some Very Good Reason, that's a lot of technical debt to maintain. If you compile mod_fcgid from source, you can arbitrarily update the config values found in fcgid_conf.h. Frustratingly, much of this config is exposed at runtime via config file directives... but not these.
Happy Coding
If you found this post because environment variables were mysteriously missing from your fcgid application, now you know why. Thanks for dropping by!