Discussion:
SSL handshake error from offlineimap when using systemd to initialize
(too old to reply)
Yubin Ruan
2018-01-21 11:12:14 UTC
Permalink
Raw Message
Hi,

I use offlineimap to synchronize my emails. I want it to do a synchronization
at system startup so recently I add a systemd service for it. However I always
get error like this:

EOF occurred in violation of protocol (_ssl.c:590)
*** Finished account 'BLACK' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'BLACK'
EOF occurred in violation of protocol (_ssl.c:590)
Traceback:
File "/usr/share/offlineimap/offlineimap/accounts.py", line 263, in syncrunner
self.__sync()
File "/usr/share/offlineimap/offlineimap/accounts.py", line 326, in __sync
remoterepos.getfolders()
File "/usr/share/offlineimap/offlineimap/repository/IMAP.py", line 384, in getfolders
imapobj = self.imapserver.acquireconnection()
File "/usr/share/offlineimap/offlineimap/imapserver.py", line 483, in acquireconnection
tls_level=self.tlslevel,
File "/usr/share/offlineimap/offlineimap/imaplibutil.py", line 186, in __init__
super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/imaplib2.py", line 2138, in __init__
IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
File "/usr/lib/python2.7/dist-packages/imaplib2.py", line 357, in __init__
self.open(host, port)
File "/usr/share/offlineimap/offlineimap/imaplibutil.py", line 194, in open
super(WrappedIMAP4_SSL, self).open(host, port)
File "/usr/lib/python2.7/dist-packages/imaplib2.py", line 2151, in open
self.ssl_wrap_socket()
File "/usr/lib/python2.7/dist-packages/imaplib2.py", line 522, in ssl_wrap_socket
self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ca_certs=self.ca_certs, cert_reqs=cert_reqs, ssl_version=ssl_version)
File "/usr/lib/python2.7/ssl.py", line 933, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 601, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 830, in do_handshake
self._sslobj.do_handshake()

Currently I don't know what the problem is, but:

1. usually (after system startup) the same service is invoked by a timer
and it works well so there is no problem with the script.

2. I believe the network is reachable, because the system will
auto-connect WIFI after system startup. Maybe the initialization order is
not configured properly? If so please see my mail service file below.

I heard that to perform a SSL handshake the system have to contain some
randomness (such that some random keys can be generated), so maybe a SSL
handshake at system startup is deemed to fail?

FYI, this is my systemd mail service file:

# ~/.config/systemd/user/mail.service
# this file is enabled with "systemctl --user enable mail.service
[Unit]
Description=Sync mail
After=network.target

[Service]
Type=oneshot
ExecStart=/path/to/the/script/mmail
TimeoutStartSec=1min30s

[Install]
WantedBy=default.target

--
Yubin
Lennart Poettering
2018-01-22 12:54:36 UTC
Permalink
Raw Message
Post by Yubin Ruan
Hi,
I use offlineimap to synchronize my emails. I want it to do a synchronization
at system startup so recently I add a systemd service for it. However I always
EOF occurred in violation of protocol (_ssl.c:590)
This suggests your network doesn't work when you invoke this.
Post by Yubin Ruan
1. usually (after system startup) the same service is invoked by a timer
and it works well so there is no problem with the script.
2. I believe the network is reachable, because the system will
auto-connect WIFI after system startup. Maybe the initialization order is
not configured properly? If so please see my mail service file below.
Well, this is necessarily racy: your network setup races agains your
offlineimap invocation...
Post by Yubin Ruan
I heard that to perform a SSL handshake the system have to contain some
randomness (such that some random keys can be generated), so maybe a SSL
handshake at system startup is deemed to fail?
I doubt this is related.

Lennart
--
Lennart Poettering, Red Hat
Yubin Ruan
2018-01-23 18:05:20 UTC
Permalink
Raw Message
Post by Lennart Poettering
Post by Yubin Ruan
Hi,
I use offlineimap to synchronize my emails. I want it to do a synchronization
at system startup so recently I add a systemd service for it. However I always
EOF occurred in violation of protocol (_ssl.c:590)
This suggests your network doesn't work when you invoke this.
Post by Yubin Ruan
1. usually (after system startup) the same service is invoked by a timer
and it works well so there is no problem with the script.
2. I believe the network is reachable, because the system will
auto-connect WIFI after system startup. Maybe the initialization order is
not configured properly? If so please see my mail service file below.
Well, this is necessarily racy: your network setup races agains your
offlineimap invocation...
I got in the configuration file

[Unit]
After=network.target

Isn't this enough to get the initialization order right?

--
Yubin
Reindl Harald
2018-01-23 08:09:50 UTC
Permalink
Raw Message
Post by Yubin Ruan
Post by Lennart Poettering
Well, this is necessarily racy: your network setup races agains your
offlineimap invocation...
I got in the configuration file
[Unit]
After=network.target
Isn't this enough to get the initialization order right?
no, the target does more or less nothing usefull

depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is
called in detail, i don#t use it)
Lennart Poettering
2018-01-23 15:10:10 UTC
Permalink
Raw Message
Post by Reindl Harald
Post by Yubin Ruan
I got in the configuration file
[Unit]
After=network.target
Isn't this enough to get the initialization order right?
no, the target does more or less nothing useful
Well, it doesn't do what Yubin assumes it does, but it certainly does
something "useful": while it doesn't make the network *connectivity*
is up, it does make sure the network *subsystem* is. And that does have
uses: during shutdown it makes sure that your service is terminated
before the network subsystem goes away. During start-up otoh it is
indeed with little effect usually.
Post by Reindl Harald
depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is called in
detail, i don#t use it)
Nope. Use "network-online.target" if you are looking for a generic
unit to order after that is reached only after the network has been
"configured" for the first time, for some vague definition of
"configured", that is up to the networking implementation to fill with
sense...

Lennart
--
Lennart Poettering, Red Hat
Yubin Ruan
2018-01-24 07:13:36 UTC
Permalink
Raw Message
Post by Lennart Poettering
Post by Reindl Harald
Post by Yubin Ruan
I got in the configuration file
[Unit]
After=network.target
Isn't this enough to get the initialization order right?
no, the target does more or less nothing useful
Well, it doesn't do what Yubin assumes it does, but it certainly does
something "useful": while it doesn't make the network *connectivity*
is up, it does make sure the network *subsystem* is. And that does have
uses: during shutdown it makes sure that your service is terminated
before the network subsystem goes away. During start-up otoh it is
indeed with little effect usually.
Post by Reindl Harald
depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is called in
detail, i don#t use it)
Nope. Use "network-online.target" if you are looking for a generic
unit to order after that is reached only after the network has been
"configured" for the first time, for some vague definition of
"configured", that is up to the networking implementation to fill with
sense...
Now I have these in the configuration file

[Unit]
Description=Sync mail
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=oneshot
ExecStart=/path/to/the/script
TimeoutStartSec=1min30s

[Install]
WantedBy=default.target

However the script is still broken at system startup. Hmm...I am using a
Ubuntu 16.04LTS. I will post if there are any news.

--
Yubin
Reindl Harald
2018-01-24 07:57:18 UTC
Permalink
Raw Message
Post by Yubin Ruan
Post by Lennart Poettering
Post by Reindl Harald
depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is called in
detail, i don#t use it)
Nope. Use "network-online.target" if you are looking for a generic
unit to order after that is reached only after the network has been
"configured" for the first time, for some vague definition of
"configured", that is up to the networking implementation to fill with
sense...
Now I have these in the configuration file
[Unit]
Description=Sync mail
Wants=network-online.target
After=network.target network-online.target
[Service]
Type=oneshot
ExecStart=/path/to/the/script
TimeoutStartSec=1min30s
[Install]
WantedBy=default.target
However the script is still broken at system startup. Hmm...I am using a
Ubuntu 16.04LTS. I will post if there are any news
AGAIN: how is your network started

Lennart is *not* correct - at least on Fedora all the wait-online stuff
don't work while "After=network.service" does when you still ue the
cliassic network.service for a lot of obvious reasons

[***@srv-rhsoft:~]$ cat /etc/rc.d/init.d/network
#! /bin/bash
#
# network Bring up/down networking
#
# chkconfig: - 10 90
# description: Activates/Deactivates all network interfaces configured to \
# start at boot time.
#
### BEGIN INIT INFO
# Provides: $network
# Should-Start: iptables ip6tables NetworkManager-wait-online
NetworkManager $network-pre
# Short-Description: Bring up/down networking
# Description: Bring up/down networking
### END INIT INFO
Yubin Ruan
2018-01-24 08:59:13 UTC
Permalink
Raw Message
Post by Reindl Harald
Post by Yubin Ruan
Post by Lennart Poettering
Post by Reindl Harald
depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is called in
detail, i don#t use it)
Nope. Use "network-online.target" if you are looking for a generic
unit to order after that is reached only after the network has been
"configured" for the first time, for some vague definition of
"configured", that is up to the networking implementation to fill with
sense...
Now I have these in the configuration file
[Unit]
Description=Sync mail
Wants=network-online.target
After=network.target network-online.target
[Service]
Type=oneshot
ExecStart=/path/to/the/script
TimeoutStartSec=1min30s
[Install]
WantedBy=default.target
However the script is still broken at system startup. Hmm...I am using a
Ubuntu 16.04LTS. I will post if there are any news
AGAIN: how is your network started
Lennart is *not* correct - at least on Fedora all the wait-online stuff
don't work while "After=network.service" does when you still ue the cliassic
network.service for a lot of obvious reasons
#! /bin/bash
#
# network Bring up/down networking
#
# chkconfig: - 10 90
# description: Activates/Deactivates all network interfaces configured to \
# start at boot time.
#
### BEGIN INIT INFO
# Provides: $network
# Should-Start: iptables ip6tables NetworkManager-wait-online NetworkManager
$network-pre
# Short-Description: Bring up/down networking
# Description: Bring up/down networking
### END INIT INFO
Below are /etc/init.d/networking and /etc/init.d/network-manager respectively.
It seems that it is /etc/init.d/networking that is responsible for bringing up
the network.

######################################
# /etc/init.d/networking
######################################
#!/bin/sh -e
### BEGIN INIT INFO
# Provides: networking ifupdown
# Required-Start: mountkernfs $local_fs urandom
# Required-Stop: $local_fs
# Default-Start: S
# Default-Stop: 0 6
# Short-Description: Raise network interfaces.
# Description: Prepare /run/network directory, ifstate file and raise network interfaces, or take them down.
### END INIT INFO

PATH="/sbin:/bin"
RUN_DIR="/run/network"
IFSTATE="$RUN_DIR/ifstate"
STATEDIR="$RUN_DIR/state"

[ -x /sbin/ifup ] || exit 0
[ -x /sbin/ifdown ] || exit 0

. /lib/lsb/init-functions

CONFIGURE_INTERFACES=yes
EXCLUDE_INTERFACES=
VERBOSE=no

[ -f /etc/default/networking ] && . /etc/default/networking

verbose=""
[ "$VERBOSE" = yes ] && verbose=-v

process_exclusions() {
set -- $EXCLUDE_INTERFACES
exclusions=""
for d
do
exclusions="-X $d $exclusions"
done
echo $exclusions
}

process_options() {
[ -e /etc/network/options ] || return 0
log_warning_msg "/etc/network/options still exists and it will be IGNORED! Please use /etc/sysctl.conf instead."
}

check_ifstate() {
if [ ! -d "$RUN_DIR" ] ; then
if ! mkdir -p "$RUN_DIR" ; then
log_failure_msg "can't create $RUN_DIR"
exit 1
fi
if ! chown root:netdev "$RUN_DIR" ; then
log_warning_msg "can't chown $RUN_DIR"
fi
fi
if [ ! -r "$IFSTATE" ] ; then
if ! :> "$IFSTATE" ; then
log_failure_msg "can't initialise $IFSTATE"
exit 1
fi
fi
}

check_network_file_systems() {
[ -e /proc/mounts ] || return 0

if [ -e /etc/iscsi/iscsi.initramfs ]; then
log_warning_msg "not deconfiguring network interfaces: iSCSI root is mounted."
exit 0
fi

while read DEV MTPT FSTYPE REST; do
case $DEV in
/dev/nbd*|/dev/nd[a-z]*|/dev/etherd/e*|curlftpfs*)
log_warning_msg "not deconfiguring network interfaces: network devices still mounted."
exit 0
;;
esac
case $FSTYPE in
nfs|nfs4|smbfs|ncp|ncpfs|cifs|coda|ocfs2|gfs|pvfs|pvfs2|fuse.httpfs|fuse.curlftpfs)
log_warning_msg "not deconfiguring network interfaces: network file systems still mounted."
exit 0
;;
esac
done < /proc/mounts
}

check_network_swap() {
[ -e /proc/swaps ] || return 0

while read DEV MTPT FSTYPE REST; do
case $DEV in
/dev/nbd*|/dev/nd[a-z]*|/dev/etherd/e*)
log_warning_msg "not deconfiguring network interfaces: network swap still mounted."
exit 0
;;
esac
done < /proc/swaps
}

ifup_hotplug () {
if [ -d /sys/class/net ]
then
ifaces=$(for iface in $(ifquery --list --allow=hotplug)
do
link=${iface##:*}
link=${link##.*}
if [ -e "/sys/class/net/$link" ]
then
# link detection does not work unless we up the link
ip link set "$iface" up || true
if [ "$(cat /sys/class/net/$link/operstate)" = up ]
then
echo "$iface"
fi
fi
done)
if [ -n "$ifaces" ]
then
ifup $ifaces "$@" || true
fi
fi
}

case "$1" in
start)
if init_is_upstart; then
exit 1
fi
process_options
check_ifstate

if [ "$CONFIGURE_INTERFACES" = no ]
then
log_action_msg "Not configuring network interfaces, see /etc/default/networking"
exit 0
fi
set -f
exclusions=$(process_exclusions)
log_action_begin_msg "Configuring network interfaces"
if [ -x /sbin/udevadm ]; then
if [ -n "$(ifquery --list --exclude=lo)" ] || [ -n "$(ifquery --list --allow=hotplug)" ]; then
udevadm settle || true
fi
fi
if ifup -a $exclusions $verbose && ifup_hotplug $exclusions $verbose
then
log_action_end_msg $?
else
log_action_end_msg $?
fi
;;

stop)
if init_is_upstart; then
exit 0
fi
check_network_file_systems
check_network_swap

log_action_begin_msg "Deconfiguring network interfaces"
if ifdown -a --exclude=lo $verbose; then
log_action_end_msg $?
else
log_action_end_msg $?
fi
;;

reload)
if init_is_upstart; then
exit 1
fi
process_options

log_action_begin_msg "Reloading network interfaces configuration"
state=$(ifquery --state)
ifdown -a --exclude=lo $verbose || true
if ifup --exclude=lo $state $verbose ; then
log_action_end_msg $?
else
log_action_end_msg $?
fi
;;

force-reload|restart)
if init_is_upstart; then
exit 1
fi
process_options

log_warning_msg "Running $0 $1 is deprecated because it may not re-enable some interfaces"
log_action_begin_msg "Reconfiguring network interfaces"
ifdown -a --exclude=lo $verbose || true
set -f
exclusions=$(process_exclusions)
if ifup -a --exclude=lo $exclusions $verbose && ifup_hotplug $exclusions $verbose
then
log_action_end_msg $?
else
log_action_end_msg $?
fi
;;

*)
echo "Usage: /etc/init.d/networking {start|stop|reload|restart|force-reload}"
exit 1
;;
esac

exit 0

########################################
# /etc/init.d/network-manager
########################################
#! /bin/sh
### BEGIN INIT INFO
# Provides: network-manager
# Required-Start: $remote_fs dbus udev
# Required-Stop: $remote_fs dbus udev
# Should-Start: $syslog
# Should-Stop: $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: network connection manager
# Description: Daemon for automatically switching network
# connections to the best available connection.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="network connection manager"
NAME="NetworkManager"

DAEMON=/usr/sbin/$NAME

PIDFILE=/var/run/$NAME/$NAME.pid

SCRIPTNAME=/etc/init.d/network-manager

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

. /lib/lsb/init-functions

test -f /etc/default/NetworkManager && . /etc/default/NetworkManager

#
# Function that starts the daemon/service.
#
d_start() {
start-stop-daemon --start --quiet --pidfile $PIDFILE \
--exec $DAEMON -- $DAEMON_OPTS
}

#
# Function that stops the daemon/service.
#
d_stop() {
start-stop-daemon --stop --retry 5 --quiet --pidfile $PIDFILE \
--exec $DAEMON
}


case "$1" in
start)
log_daemon_msg "Starting $DESC" "$NAME"
d_start
case "$?" in
0) log_end_msg 0 ;;
1) log_progress_msg "already started"
log_end_msg 0 ;;
*) log_end_msg 1 ;;
esac
;;
stop)
log_daemon_msg "Stopping $DESC" "$NAME"
d_stop
case "$?" in
0) log_end_msg 0 ;;
1) log_progress_msg "already stopped"
log_end_msg 0 ;;
*) log_end_msg 1 ;;
esac
;;
restart|force-reload)
$0 stop
$0 start
;;
status)
status_of_proc -p $PIDFILE $DAEMON $NAME && exit 0 || exit $?
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload|status}" >&2
exit 1
;;
esac

exit 0

--
Yubin
Reindl Harald
2018-01-24 09:06:06 UTC
Permalink
Raw Message
Post by Yubin Ruan
Post by Reindl Harald
Post by Yubin Ruan
Post by Lennart Poettering
Post by Reindl Harald
depeding on how your network is configured use "network.service" or
"networkmanager.service" (or however the networkmanager service is called in
detail, i don#t use it)
Nope. Use "network-online.target" if you are looking for a generic
unit to order after that is reached only after the network has been
"configured" for the first time, for some vague definition of
"configured", that is up to the networking implementation to fill with
sense...
Now I have these in the configuration file
[Unit]
Description=Sync mail
Wants=network-online.target
After=network.target network-online.target
[Service]
Type=oneshot
ExecStart=/path/to/the/script
TimeoutStartSec=1min30s
[Install]
WantedBy=default.target
However the script is still broken at system startup. Hmm...I am using a
Ubuntu 16.04LTS. I will post if there are any news
AGAIN: how is your network started
Lennart is *not* correct - at least on Fedora all the wait-online stuff
don't work while "After=network.service" does when you still ue the cliassic
network.service for a lot of obvious reasons
well, then try "After=networking.service network-manager.service"
despite what others saying about targets - iam dong the same with
"After=network.service" on Fedora for some years on 30 production
servers and it works just fine

in fact we have "After=network.service systemd-networkd.service
network-online.target" on any service which needs networking on Fedora
and RHEL7
Post by Yubin Ruan
Below are /etc/init.d/networking and /etc/init.d/network-manager respectively.
It seems that it is /etc/init.d/networking that is responsible for bringing up
the network.
######################################
# /etc/init.d/networking
######################################
########################################
# /etc/init.d/network-manager
########################################
Michael Chapman
2018-01-23 08:35:47 UTC
Permalink
Raw Message
Post by Yubin Ruan
Post by Lennart Poettering
Post by Yubin Ruan
Hi,
I use offlineimap to synchronize my emails. I want it to do a synchronization
at system startup so recently I add a systemd service for it. However I always
EOF occurred in violation of protocol (_ssl.c:590)
This suggests your network doesn't work when you invoke this.
Post by Yubin Ruan
1. usually (after system startup) the same service is invoked by a timer
and it works well so there is no problem with the script.
2. I believe the network is reachable, because the system will
auto-connect WIFI after system startup. Maybe the initialization order is
not configured properly? If so please see my mail service file below.
Well, this is necessarily racy: your network setup races agains your
offlineimap invocation...
I got in the configuration file
[Unit]
After=network.target
Isn't this enough to get the initialization order right?
No, network.target is mostly about ordering things correctly during
shutdown.

You need to do two things:

* Use After=network-online.target in your unit.
* Enable some _other_ service that detects when the network is "online"
(whatever that means), and that is ordered Before=network-online.target.

If you are using systemd-networkd, for instance, this service is
systemd-networkd-wait-online.service. If you are using NetworkManager, you
want NetworkManager-wait-online.service.

See https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for
further details.
Cristian Rodríguez
2018-01-22 13:28:06 UTC
Permalink
Raw Message
Post by Yubin Ruan
Hi,
I use offlineimap to synchronize my emails. I want it to do a synchronization
at system startup so recently I add a systemd service for it. However I always
EOF occurred in violation of protocol (_ssl.c:590)
Socket was closed but not the SSL session.. not a systemd problem..
Post by Yubin Ruan
1. usually (after system startup) the same service is invoked by a timer
and it works well so there is no problem with the script.
It is racing against initial network setup.. once the network settles it
works as expected.
Post by Yubin Ruan
2. I believe the network is reachable, because the system will
auto-connect WIFI after system startup. Maybe the initialization order is
not configured properly? If so please see my mail service file below.
You may want to order your services after network-online and enable the
systemd-network-online service.. however that may still race.
Post by Yubin Ruan
I heard that to perform a SSL handshake the system have to contain some
randomness (such that some random keys can be generated),
Correct, but any of the ssl libraries in linux will inmediately return
or terminate the process in case of a entropy failure, because such
failure is fatal and the whole security of the ssl session is screwed.
Loading...