Nginx 500 error – “/var/lib/nginx/tmp/fastcgi/*” failed (13: Permission denied)
Today I came across an interesting error with our Web servers at work. Out of the blue, users were all of a sudden experiencing Nginx 500 error when trying to save Magento Products / Category and even exporting product data (default export tool).
What’s interesting about this error is that it only started occurring recently and was only affected by certain products or categories.
When you start to analyse the nginx error / access logs you will see that your request (usually a html post) would result in a Permission denied, see the snippet below:
2017/10/06 13:34:01 [crit] 20954#0: *18653 open() “/var/lib/nginx/tmp/fastcgi/3/01/0000000013” failed (13: Permission denied) while reading upstream, client: 127.0.0.1, server: www.example.com, request: “POST /index.php/admin/export/export/key/d01555441b5887176df57427347a0a70/entity/catalog_product/file_format/csv HTTP/1.1”, upstream: “fastcgi://unix:/var/run/php-fpm/php-fpm.sock:”, host: “www.example.com”, referrer: “https://www.example.com/index.php/admin/export/index/key/b9cecec7cec6ef2b65306d82911d5xsrcd6d/”
The first question is why did this happen all of a sudden?
Not sure completely. After speaking to the server hosts, no server updates were made, though I’m not sure if nginx auto-updated and changed permissions.
Apparently when nginx needs to write temporary data to files, it does to in the nginx tmp directory – and by default unless you specify otherwise in your nginx.conf file, this could be located here: /var/lib/nginx/tmp/
So lets view the permissions of that folder – based on what’s reported in our nginx error log file (as per the snippet above).
gurdeep@random-server: ls -lath /var/lib/nginx/tmp drwx------. 7 nginx nginx 4096 Sep 18 10:20 /var/lib/nginx/tmp
We can see that the owner and group of the above folder is “nginx:nginx”.
What that is saying the user nginx and whatever processes it owns, will be able to modify the contents of those folders. Let’s verify if our nginx process is indeed running as the user nginx
gurdeep@random-server: ps aux | grep nginx root 13895 0.0 0.0 126996 5912 ? Ss 19:07 0:00 nginx: master process /usr/sbin/nginx gurdeep 13896 0.0 0.0 128712 10464 ? S 19:07 0:01 nginx: worker process gurdeep 13897 0.0 0.0 128680 10312 ? S 19:07 0:01 nginx: worker process gurdeep 13898 0.0 0.0 128744 10312 ? S 19:07 0:01 nginx: worker process
Hmm! According to the output (left side), the user gurdeep owns and is running the nginx worker processes! Not nginx!
But… maybe the user gurdeep is a member of the nginx group and can access these folders? We can check the user’s groups by using the groups command:
gurdeep@random-server: groups gurdeep gurdeep : www-data sudo
So clearly the user gurdeep is not a member of the nginx group so definitely doesn’t have write permissions (I’m glazing over chmod permissions btw).
What about nginx.conf?
I’m glad you ask. In most cases, the default user the nginx processes run as is declared in the nginx.conf file. So open it up and have a look for a similar looking line as below:
#nginx.conf user gurdeep; worker_processes auto; pid /run/nginx.pid;
Look for the line that begins with user, in the example above it’s user gurdeep;
This means nginx will spawn processes by the user gurdeep. You may have something different like www-user, ec1-blah etc.
So based on our findings above, we basically need to make sure the owner of the nginx workers / processes has permission to the folder that is throwing errors in the log files:
gurdeep@random-server: sudo chown gurdeep:root /var/lib/nginx/tmp
However that did not resolve the problem.. I researched into this a little further and apparently you also need to set the correct owner of the parent directory /var/lib/nginx/:
gurdeep@random-server: sudo chown gurdeep:root /var/lib/nginx
Restart Nginx and your troubles should dissappear:
Note: It’s worth mentioning I’ve purposely set the group to :root, but I’ve seen people specify the same group based on their php-fpm process as they’ve also used the same user for the nginx workers (i.e: www-data):
gurdeep@random-server: sudo chown www-data:www-data /var/lib/nginx gurdeep@random-server: sudo chown www-data:www-data /var/lib/nginx/tmp