Running multiple instances of varnish using systemd

Running multiple instances of varnish using systemd

If you have not yet found a complete solution for Running multiple instances of varnish using systemd here are the steps you need to do get this done working.
In this Tutorial we will add multi instances support systemd service script for following varnish daemons: varnishd, varnishncsa, varnishlog, varnish-agent

Prerequisites:

• A working Varnish 4.0+ Setup
• Debian 8.0+ like Operating System with Systemd
• Texteditor (vi, joe, nano) of your choice

1. Lets start over creating default files which contains our customized settings which our service script will load

Goto: /etc/default and first copy the predefined daemon option templates to your target instancename:

cd /etc/default
cp -a varnish varnish-yourinstancename
cp -a varnish-agent varnish-agent-yourinstancename

Note: Customized varnishnca, varnishlog default files arent necessary, since we only control them by the -n parameter in the service script later on.

2. Change your default files that they fit your setup
I will give you one example for: /etc/default/varnish-yourinstancename

# /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and $MEMLOCK
# Should we start varnishd at boot?  Set to "no" to disable.
START=yes

# Maximum number of open files (for ulimit -n)
NFILES=131072

# Maximum locked memory size (for ulimit -l)
# Used for locking the shared memory log in memory.  If you increase log size,
# you need to increase this number as well
MEMLOCK=82000

DAEMON_OPTS="-a 10.1.0.1:80 -a 127.1.0.1:80 
             -T localhost:6083 
             -f /etc/varnish/yourinstancename.vcl 
             -p thread_pool_add_delay=2 
             -p thread_pools=2 
             -p thread_pool_min=200 
             -p thread_pool_max=4000 
             -p syslog_cli_traffic=off 
             -t 120 
             -S /etc/varnish/secret 
             -s malloc,256M"

Note: Watch for the vcl filename, this is important to match the instancename you choose for later service start usage.
Also keep in mind that you need to update the IP and Ports for each default file, so they wont overlap with other instances.

If you need Varnish access logging in Apache format here is the varnishncsma example: /etc/default/varnishncsa-yourinstancename

# Configuration file for varnishncsa
#
# Note: If systemd is installed, this file is obsolete and ignored.  You will
# need to copy /lib/systemd/system/varnishncsa.service to /etc/systemd/system/
# and edit that file.
#
# Uncomment this to enable logging for varnish.  Please make sure you have
# enough disk space for significant amounts of log data.  To disable logging,
# set the variable to "0", "no", or leave it unset.
#
# NCSA log format, to be used by HTTP log analyzers
VARNISHNCSA_ENABLED=1

DAEMON_OPTS="-a -w /var/log/varnish/yourinstancename.log -F '%{Host}I %h %l %u %t "%m %U %H" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:time_firstbyte}x'"

And if you also running varnish-agent and varnish dashboard: /etc/default/varnish-agent-yourinstancename

# Configuration file for Varnish Agent.
#
# /etc/init.d/varnish-agent expects the variable $DAEMON_OPTS to be
# set from this shell script fragment.
#
# See varnish-agent(1) for an overview of valid settings.

DAEMON_OPTS="-T localhost:6083 -c 6087 -q -H /var/www/html/varnish-dashboard"

# an example of vac callback
# DAEMON_OPTS="-z http://vac_server/api/rest/register"

Note: Make sure that the -T variable Host and Port is the same value than the one from your previous varnish-yourinstancename defaults file.
Give the agent a listening Port on the -c variable under that you will reach the dashboard later on.

2. Now we will define custom made service scripts to run our Varnish instances

We will move to our /etc/systemd/system directory which is our safe place to store systemd scripts preventing them from overwriting when upgrading our packages.

Lets start over with creating new service for varnishd.

Create a new file called: /etc/systemd/system/varnish@.service and put following content into it:

# Multi instance version of Varnish by Jules [https://ispire.me]
#
# create config file: /etc/default/varnish-{instancename}
# create vcl file: /etc/varnish/{instancename}.vcl
#
# run: systemctl start varnish@{instancename}

[Unit]
Description=Varnish HTTP accelerator Multi Instance

ConditionPathExists=/etc/default/varnish-%I

[Service]
EnvironmentFile=/etc/default/varnish-%I
Type=forking
LimitNOFILE=131072
LimitMEMLOCK=82000
Restart=always
RestartSec=5
ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/%I.vcl
ExecStart=/usr/sbin/varnishd -n %I $DAEMON_OPTS
ExecReload=/usr/share/varnish/reload-vcl

[Install]
WantedBy=multi-user.target

Now create /etc/systemd/system/varnishlog@.service with content:

# Multi instance version of Varnishlog by Jules [https://ispire.me]
#
# create config file: /etc/default/varnish-{instancename}
# create vcl file: /etc/varnish/{instancename}.vcl
#
# run: systemctl start varnishlog@{instancename}

[Unit]
Description=Varnish HTTP accelerator log daemon Multi Instance

ConditionPathExists=/etc/default/varnish-%I

After=varnish@%I.service

[Service]
User=varnishlog
Restart=always
RestartSec=5
ExecStart=/usr/bin/varnishlog -a -n %I -t off -w /var/log/varnish/%I.log
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Same thing for /etc/systemd/system/varnishncsa@.service with content:

# Multi instance version of Varnishncsa by Jules [https://ispire.me]
#
# create config file: /etc/default/varnish-{instancename}
# create vcl file: /etc/varnish/{instancename}.vcl
#
# run: systemctl start varnishncsa@{instancename}

[Unit]
Description=Varnish HTTP accelerator log daemon Multi instance

ConditionPathExists=/etc/default/varnishncsa-%I

After=varnish@%I.service

[Service]
User=varnishlog
Restart=always
RestartSec=5
EnvironmentFile=/etc/default/varnishncsa/varnishncsa-%I
ExecStart=/usr/bin/varnishncsa -n %I -t off $DAEMON_OPTS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

And finally the /etc/systemd/system/varnish-agent@.service with content:

# Multi instance version of Varnish-agent by Jules [https://ispire.me]
#
# create config file: /etc/default/varnish-agent-{instancename}
# create vcl file: /etc/varnish/{instancename}.vcl
#
# run: systemctl start varnish-agent@{instancename}

[Unit]
Description=Varnish Agent Multi Instance

ConditionPathExists=/etc/default/varnish-agent-%I

After=varnish@%I.service

[Service]
EnvironmentFile=/etc/default/varnish-agent-%I
Type=forking
Restart=always
RestartSec=5
PIDFile=/var/run/varnish-agent.pid
ExecStart=/usr/bin/varnish-agent 
        -P /var/run/varnish-agent.pid 
        -n %I $DAEMON_OPTS

[Install]
WantedBy=multi-user.target

After we have created all our service scripts, here comes the next step

3. Activating our new created systemd service scripts

Lets activate the services so they will automatically start on Server boot:

systemctl enable varnish@yourinstancename
systemctl enable varnishlog@yourinstancename
systemctl enable varnishncsa@yourinstancename
systemctl enable varnish-agent@yourinstancename

And finally lets start them:

systemctl start varnish@yourinstancename
systemctl start varnishlog@yourinstancename
systemctl start varnishncsa@yourinstancename
systemctl start varnish-agent@yourinstancename

Note: Only enable/start varnishlog if needed for debugging purpose. It used to write a lot data in Live environments.

See, with this method you can run as many separate Varnish instances as you want by just defining and instance name behind the @.

Cheers!