Discussion:
Socket activation (for GitWeb)?
Add Reply
Alex Ivanov
2018-04-04 19:10:02 UTC
Reply
Permalink
Raw Message
Hi.
I want to use systemd as fastcgi spawner for gitweb + nginx.
The traffic is low and number of users is limited + traversal bots. For that reason I've decided to use following mimimal services

gitweb.socket:
[Unit]
Description=GitWeb Socket

[Socket]
ListenStream=/run/gitweb.sock
Accept=false

[Install]
WantedBy=sockets.target

gitweb.service:
[Unit]
Description=GitWeb Service

[Service]
Type=simple
ExecStart=/path/to/gitweb.cgi --fcgi
StandardInput=socket

However this scheme is not resistant to simple DDOS.
E.g. traversal bots often kill the service by opening non existing path (e.g http://host/?p=repo;a=blob;f=nonexisting/path;hb=HEAD showing in browser 404 - Cannot find file) many times consecutively, which leads to
Apr 03 21:32:10 host systemd[1]: gitweb.service: Start request repeated too quickly.
Apr 03 21:32:10 host systemd[1]: gitweb.service: Failed with result 'start-limit-hit'.
Apr 03 21:32:10 host systemd[1]: Failed to start GitWeb service.
in journal and 502 Bad Gateway in browser.

Could someone please show me how to correct this issue?
Mantas Mikulėnas
2018-04-04 20:38:32 UTC
Reply
Permalink
Raw Message
Post by Alex Ivanov
Hi.
I want to use systemd as fastcgi spawner for gitweb + nginx.
The traffic is low and number of users is limited + traversal bots. For
that reason I've decided to use following mimimal services
[Unit]
Description=GitWeb Socket
[Socket]
ListenStream=/run/gitweb.sock
Accept=false
[Install]
WantedBy=sockets.target
[Unit]
Description=GitWeb Service
[Service]
Type=simple
ExecStart=/path/to/gitweb.cgi --fcgi
StandardInput=socket
However this scheme is not resistant to simple DDOS.
E.g. traversal bots often kill the service by opening non existing path
(e.g http://host/?p=repo;a=blob;f=nonexisting/path;hb=HEAD showing in
browser 404 - Cannot find file) many times consecutively, which leads to
Apr 03 21:32:10 host systemd[1]: gitweb.service: Start request repeated too quickly.
Apr 03 21:32:10 host systemd[1]: gitweb.service: Failed with result 'start-limit-hit'.
Apr 03 21:32:10 host systemd[1]: Failed to start GitWeb service.
in journal and 502 Bad Gateway in browser.
Could someone please show me how to correct this issue?
I don't quite understand how gitweb.cgi is designed and how it's supposed
to behave here.

You use Accept=no, which implies that gitweb.cgi should start *once* and
keep accepting connections in a loop forever. So why does it keep exiting
and restarting *at all*? Why does a simple 404 make it exit the whole event
loop?

If it doesn't accept connections on its own, but needs to be passed in an
already-accepted connection, that'd be Accept=yes... And I'm not sure if
systemd activation like this is suitable for very high request rates.
Post by Alex Ivanov
--
Mantas Mikulėnas <***@gmail.com>
Sent from my phone
Lennart Poettering
2018-04-05 08:55:37 UTC
Reply
Permalink
Raw Message
Post by Alex Ivanov
Hi.
I want to use systemd as fastcgi spawner for gitweb + nginx.
The traffic is low and number of users is limited + traversal bots. For that reason I've decided to use following mimimal services
[Unit]
Description=GitWeb Socket
[Socket]
ListenStream=/run/gitweb.sock
Accept=false
[Install]
WantedBy=sockets.target
[Unit]
Description=GitWeb Service
[Service]
Type=simple
ExecStart=/path/to/gitweb.cgi --fcgi
StandardInput=socket
However this scheme is not resistant to simple DDOS.
E.g. traversal bots often kill the service by opening non existing path (e.g http://host/?p=repo;a=blob;f=nonexisting/path;hb=HEAD showing in browser 404 - Cannot find file) many times consecutively, which leads to
Apr 03 21:32:10 host systemd[1]: gitweb.service: Start request repeated too quickly.
Apr 03 21:32:10 host systemd[1]: gitweb.service: Failed with result 'start-limit-hit'.
Apr 03 21:32:10 host systemd[1]: Failed to start GitWeb service.
in journal and 502 Bad Gateway in browser.
Could someone please show me how to correct this issue?
So, most socket activated services tend to stick around for a while
after processing the last request before exiting, so that connection
bursts are nicely handled. If your service exits immediately after
processing a connection then you increase the load unnecessarily, as
this means that you pay for your service's startup and shutdown phase
on every single connection all the time.

systemd will try to protect you from such "overloading" behaviour by
default, and thus enforces a start limit on all services, so that a
service starting up too often and thus consuming resources too heavily
within a short period of time is blocked. This is what you are seeing
above: the protection works, you get the "start-limit-hit" service
result.

With StartLimitIntervalSec= and StartLimitBurst= in the service unit
file you can tweak the behaviour of the limiter. You can bump up
StartLimitBurst= drastically if you like or even turn off such
limiting altogether with that setting.

However, I'd really recommend instead looking into your service, and
see if you can instead just make it stick around for a bit before
exiting after idle. For example, if the daemon only exits after 10s of
idle, then you already should get much better behaviour that can deal
a lot better with burst connections.

Lennart
--
Lennart Poettering, Red Hat
Loading...