684 commits

Author	SHA1	Message	Date
Alexandre Iooss	3c85a2bfb2	passbolt: remove role	2021-11-28 11:13:34 +01:00
Alexandre Iooss	fd0cb811a7	postgres: fix defaults	2021-11-28 11:07:13 +01:00
Alexandre Iooss	4bd431f9c3	postgresql: variables in dict	2021-11-28 11:01:29 +01:00
Alexandre Iooss	a818fd8ed9	Rename postgresql_server to postgresql	2021-11-28 10:20:17 +01:00
Alexandre Iooss	0979370418	Add motd for most plays	2021-11-27 22:16:29 +01:00
Alexandre Iooss	14b6a68040	base: configure motd	2021-11-27 20:05:14 +01:00
Alexandre Iooss	cc6f96bbc8	borgbackup-client: call update_motd role in play	2021-11-27 20:04:05 +01:00
Alexandre Iooss	07a0429ae0	nginx: call update_motd role in play	2021-11-27 20:02:08 +01:00
Alexandre Iooss	ce04f937db	prometheus: call update_motd role in play	2021-11-27 19:20:32 +01:00
Alexandre Iooss	1009298023	borgbackup_server: call update_motd role in play	2021-11-27 19:16:24 +01:00
Alexandre Iooss	ea394a01db	prometheus-federate: call update_motd role in play	2021-11-27 19:16:11 +01:00
Alexandre Iooss	b82afd13d9	update_motd: use update_motd dict	2021-11-27 19:14:39 +01:00
Alexandre Iooss	a791cda652	grafana: move Aurore specific variables out of the role	2021-11-27 18:29:05 +01:00
Alexandre Iooss	fdfed1a05a	grafana: remove trailing lines	2021-11-27 18:17:57 +01:00
Alexandre Iooss	e2acfd4031	grafana: single quote LDAP password	2021-11-27 18:17:57 +01:00
Alexandre Iooss	c7f94b54c8	grafana: validate gpg key	2021-11-27 18:17:57 +01:00
Alexandre Iooss	aba0370c5b	Add grafana playbook and machine	2021-11-27 18:17:57 +01:00
Alexandre Iooss	3efc8179bc	logrotate: restore Debian formatting	2021-11-22 18:08:25 +01:00
Alexandre Iooss	3a56439fac	update_motd: remove become true	2021-11-22 18:03:09 +01:00
Alexandre Iooss	94b8f37302	rsyslog_common: remove become true	2021-11-22 18:02:53 +01:00
Alexandre Iooss	1392e3fe64	Remove cached motd	2021-11-22 18:01:21 +01:00
Alexandre Iooss	11b3738fcd	ldap_client: Add one extra line to follow Debian	2021-11-22 18:00:57 +01:00
Jeltz	8b54121a87	Install prometheus-node-exporter-collectors	2021-09-24 01:41:01 +02:00
Jean-Marie Mineau	5d3d965112	the service does not need to be enabled	2021-09-23 19:02:26 +02:00
Jean-Marie Mineau	73e522f0c6	add exporter on bullseye	2021-09-23 18:54:06 +02:00
pz2891	b31f9bd952	Retention time is now a file that will be copied	2021-08-21 11:25:39 +02:00
pz2891	cc3b4294ae	Kepp federated datas 4 months (120 days)	2021-08-20 19:31:04 +02:00
pz2891	f17e7f7524	Add snmp pdu password to generate config	2021-08-20 18:22:00 +02:00
pz2891	0bfc631465	Remove unused files	2021-08-20 17:00:19 +02:00
pz2891	c5e6fbcfdf	Configuration for monitoring APC PDU	2021-08-20 16:58:28 +02:00
pz2891	54b073bd02	Typo in unhealthy disk rule	2021-08-18 18:53:27 +02:00
pz2891	e6b6790f63	New rule for unhealthy disks	2021-08-13 15:24:12 +02:00
Gabriel Detraz	b7ead19d50	Remove mail from re2o bug report	2021-05-25 23:11:30 +02:00
Jeltz	bb97bca456	Increase RandomizedDelaySec when hourly = 0	2021-05-23 14:09:01 +02:00
Jeltz	9296a2ed91	Add caradoc.adm.auro.re	2021-05-23 14:02:20 +02:00
Jeltz	4f2f0ffe64	Increase swap alert threshold	2021-05-19 15:32:33 +02:00
Jeltz	c8a877282f	Add 9 & 10 for Debian distribution	2021-05-19 15:29:40 +02:00
Jeltz	c6b768e1bb	Don't run borgmatic every hour if not needed	2021-05-10 13:02:45 +02:00
jeltz	ceaf75f0ad	Merge pull request 'Use a disk assisted queue for rsyslog' (#56) from rsyslog_queues into master ... Reviewed-on: Aurore/ansible#56	2021-05-04 00:54:40 +02:00
Jeltz	b29e9c0e45	Configure a disk-assisted queue for output actions	2021-04-30 16:49:00 +02:00
Otthorn	3a600d9061	Give a name to unnamed tasks	2021-04-17 17:43:49 +02:00
pz2891	11d0b46ef0	Remove port for docker instances. Remove 'remove old files' tasks	2021-04-14 20:00:16 +02:00
pz2891	013743f910	typo in docker rules	2021-04-14 19:54:37 +02:00
pz2891	1b0bff4c51	Fix deployment and add prometheus groups for hosts	2021-04-14 19:51:47 +02:00
pz2891	fde52f2e42	Alerts repository owned by prometheus	2021-04-14 19:29:12 +02:00
pz2891	e4d2416722	fix typo	2021-04-14 19:27:13 +02:00
pz2891	226b55b0d1	Update alerts (remove instance, translations)	2021-04-14 19:10:42 +02:00
pz2891	fd5ad8d5ac	Merge branch 'prometheus_postgres_exporter' of https://gitea.auro.re/Aurore/ansible into prometheus_postgres_exporter	2021-04-12 11:10:31 +02:00
pz2891	5d9a6599e8	Fix some typos, in accordance to Solal's comments	2021-04-12 11:10:15 +02:00
pz2891	3320e3e0c6	Update the labels for the alert (make complete tenses)	2021-04-12 11:01:43 +02:00
pz2891	676cc716cf	Modify label for the alert	2021-04-12 11:00:31 +02:00
pz2891	954e3e0892	End of yaml file (bad copy/paste)	2021-04-12 10:58:59 +02:00
pz2891	8c666151d6	Merge branch 'master' into prometheus_postgres_exporter	2021-04-12 10:10:17 +02:00
pz2891	1908deee9c	fix CI	2021-04-12 10:01:39 +02:00
Jeltz	e2b1f8eae5	Allow root to connect using peer authentication	2021-04-11 22:08:11 +02:00
pz2891	6c64bb214c	fix CI	2021-04-11 22:01:21 +02:00
pz2891	764f0f106d	Install postgres exporter when it is bullseye or buster	2021-04-11 21:38:29 +02:00
pz2891	c48fe1ae17	7% rollback for the warning	2021-04-11 20:57:53 +02:00
pz2891	304437da97	Remove .save file	2021-04-11 20:56:40 +02:00
pz2891	9d18ebb7f1	Fix docker rules	2021-04-11 17:18:32 +02:00
pz2891	6775d9ecde	Add docker rules	2021-04-11 16:43:34 +02:00
pz2891	9ebdf15bb9	Splite alerts on some files	2021-04-11 15:58:35 +02:00
pz2891	dd48302585	Configure Prometheus and Prometheus federate to scrape Postgres Exporter	2021-04-10 18:01:55 +02:00
pz2891	45041be2ab	Install postgres exporter	2021-04-10 17:29:50 +02:00
jeltz	6b2bc60589	Merge branch 'master' into add_rives_vm_master	2021-04-06 19:37:57 +02:00
Jeltz	91817b324c	Increase the alert threshold for temperatures	2021-04-03 08:04:10 +02:00
Jeltz	1c3127dbbe	Add more node-exporter alerts ... Source: https://awesome-prometheus-alerts.grep.to/rules.html	2021-04-02 22:55:51 +02:00
Jeltz	f80435cb31	Differentiate alerts for servers and Wi-Fi APs	2021-04-02 21:54:38 +02:00
Jeltz	06f101527d	Use a dynamic interval for UPS output voltage alerts	2021-04-02 13:57:34 +02:00
Jeltz	83f5b35e59	Fix a filename typo	2021-04-01 18:24:21 +02:00
Jeltz	35286a661a	Change an alert description	2021-04-01 18:24:03 +02:00
Jeltz	11335a6077	Fix typo in alert description	2021-04-01 18:15:22 +02:00
Jeltz	083fc4da9a	Fix permissions on prometheus.yml	2021-04-01 18:15:09 +02:00
Jeltz	a743ce09fb	Move templates of the prometheus_federate role	2021-04-01 09:42:54 +02:00
Jeltz	bc35cd8e90	Move templates of the prometheus role	2021-04-01 09:40:22 +02:00
Jeltz	5bcc428895	Remove 'instance' from description and fix typos	2021-04-01 09:36:11 +02:00
Jeltz	eeaf0f8486	Fix syntax errors	2021-04-01 06:02:40 +02:00
Jeltz	e247aa3f70	Uniform labels for alerts	2021-04-01 05:21:08 +02:00
jeltz	424aa80d8f	Merge pull request 'Use update_motd everywhere' (#44) from use_update_motd_everywhere into master ... Reviewed-on: Aurore/ansible#44	2021-03-30 10:12:14 +02:00
Jeltz	ac05da7173	Use update_motd everywhere	2021-03-30 10:08:21 +02:00
Jeltz	dff0d9922c	Store log.adm.auro.re local logs in /var/log/remote	2021-03-30 10:06:25 +02:00
pz2891	dd274891a5	resolve conflicts	2021-03-30 09:30:06 +02:00
pz2891	2952c39f70	Fix issues for installing radius-rives (baq package for postgresql-client)	2021-03-30 09:20:31 +02:00
Jeltz	85e691a0a2	Don't store journald logs to disk ... As they are already stored on disk by rsyslog.	2021-03-30 07:46:06 +02:00
Jeltz	606df65535	Cleanup logrotate role	2021-03-30 07:45:52 +02:00
Jeltz	3030d3bfab	Fix typo: use 'Reload' instead of 'Restart'	2021-03-30 07:42:46 +02:00
Jeltz	f59d9ee6f0	WIP: add logrotate config for rsyslog-managed files	2021-03-30 06:01:43 +02:00
jeltz	6d74f04db4	Merge pull request 'Better distribution of backups over time' (#49) from backups into master ... Reviewed-on: Aurore/ansible#49	2021-03-24 02:12:53 +01:00
Jeltz	21eaeb2d42	Better distribution of backups over time	2021-03-24 02:10:11 +01:00
jeltz	789c11c3e3	Merge pull request 'Cleanup borgmatic related roles' (#47) from backups into master ... Reviewed-on: Aurore/ansible#47	2021-03-18 22:19:39 +01:00
pz2891	a1533b7efd	Fix issues for installing radius-rives (baq package for postgresql-client)	2021-03-17 20:41:46 +01:00
pz2891	f662e4bd47	Remove bullseye for radius role. Add the oid for temperature of ups	2021-03-16 21:13:45 +01:00
Jeltz	3000f46c46	Randomize borgmatic timer	2021-03-16 15:05:29 +01:00
Jeltz	8524b9fa99	Fix typo	2021-03-16 14:13:12 +01:00
Jeltz	37582abfe1	Remove useless tasks from borgmatic_client	2021-03-16 13:47:14 +01:00
Jeltz	96a498c6de	Break long lines in borgmatic.service unit	2021-03-16 13:46:46 +01:00
Jeltz	1be92bad62	Log source port for NGinx	2021-03-16 09:43:13 +01:00
Jeltz	01bca6597d	Run borgmatic every hour	2021-03-16 09:38:51 +01:00
Jeltz	21a3d5af2a	Add bullseye support in 'prometheus_node'	2021-03-15 10:50:40 +01:00
jeltz	4305a60639	Merge pull request 'Backups with borg and borgmatic' (#39) from backups into master ... Reviewed-on: Aurore/ansible#39	2021-03-15 07:53:33 +01:00
Jeltz	3f3f688da4	Use 'present' instead of 'latest' (ansible-lint)	2021-03-15 07:51:48 +01:00
Jeltz	6713b550b6	Merge branch 'master' into backups	2021-03-15 07:50:11 +01:00
Jeltz	cb3ec07121	Use 'inventory_hostname' instead of 'ansible_fqdn' ... While 'ansible_fdqn' can be changed by a compromised host, 'inventory_hostname' can't (hopefully). It should therefore no longer be possible for the said host to access the backups of another host.	2021-03-15 07:25:09 +01:00
Solal Nathan	243ec1fe9d	[borgbackup_client] VaRi0u5 f1X3s	2021-03-15 01:04:42 +01:00
Jeltz	f15b222cdc	Allow root to log as postgres	2021-03-14 23:45:36 +01:00
Solal Nathan	7480a7c565	[borgbackup_client] precedence rules and sain defaults for borg config	2021-03-14 22:02:34 +01:00
Solal Nathan	b14b359027	[borgbackup_client] add exlude path to conf	2021-03-14 19:21:15 +01:00
Solal Nathan	33a1ec02f3	[borgbackup_client] update config directory to be homogeneous	2021-03-14 19:07:02 +01:00
Solal Nathan	ebfc4f2a26	[borgbackup_client] do update cache	2021-03-14 19:03:44 +01:00
Jeltz	86f8b31159	Delegate facts for borgbackup_client	2021-03-14 18:44:13 +01:00
Jeltz	d9f1104309	Move id_remote to /etc/borgmatic	2021-03-14 18:42:26 +01:00
Solal Nathan	c6cae75031	[borgbackup_server] fix /borg permissions	2021-03-14 18:29:33 +01:00
Solal Nathan	46d10022ea	[borgbackup_client] fix rentention date to int and list correctly source directories	2021-03-14 18:24:36 +01:00
Solal Nathan	ff750c5b63	[borgbackup_client] remove 1 minute sleep and fix verbosity	2021-03-14 18:23:44 +01:00
Solal Nathan	2651432582	[WIP] various fixes	2021-03-14 18:22:52 +01:00
Solal Nathan	d928c7f7f0	[borgbackup_client] rename variable correclty	2021-03-14 16:11:40 +01:00
Solal Nathan	021a5ef1e8	[borgbackup_client] various fixes for ssh keys	2021-03-14 16:11:18 +01:00
Jeltz	c99b611b8f	Various fixes	2021-03-14 14:17:36 +01:00
Jeltz	8112788396	[borgbackup_client] Add 'user:' in authorized_key	2021-03-14 13:18:30 +01:00
Jeltz	2f2f71422f	[borgbackup_client] Move some handlers to tasks	2021-03-14 13:16:08 +01:00
Jeltz	637b74a2ad	Fix some linter issues	2021-03-13 05:05:30 +01:00
Jeltz	f45cd77510	Merge branch 'master' into logs-first-phase	2021-03-13 05:02:30 +01:00
pz2891	f6e1949c21	Adding master VM for Rives and adapt radius role for bullseye	2021-03-12 12:29:52 +01:00
Otthorn	965bbe62a4	[borgbackup_client] configure encryption passphrase and storage	2021-03-12 01:46:35 +01:00
Otthorn	3f8ffbe164	[borgbackup_client] Add borg username and group defaults	2021-03-12 00:01:11 +01:00
Otthorn	531f7593d2	[borgbackup_client] fix identation	2021-03-11 23:37:55 +01:00
Otthorn	313314a674	[borgbackup_client] fix risky file permission on apt config for pinning version	2021-03-11 23:36:27 +01:00
Otthorn	4642395330	[borgbackup_client] Add initial role defintion	2021-03-11 23:29:57 +01:00
Jeltz	f0f56ecd3f	Fix linter-related issues	2021-03-11 20:15:35 +01:00
Jeltz	db8dbb6c7a	Add borgbackup_server role	2021-03-11 20:08:41 +01:00
jeltz	2a6c2b30de	Merge pull request 'Rôle pour motd' (#38) from update_motd into master ... Reviewed-on: Aurore/ansible#38	2021-03-11 19:34:41 +01:00
pz2891	6125856c60	Merge branch 'monitoring_ups'	2021-03-11 14:06:38 +01:00
pz2891	d233fc2759	Update of threesold for warning battery	2021-03-11 13:23:15 +01:00
Jeltz	6095d9cef9	Add 'no_log' for postgres passwords	2021-03-10 18:18:08 +01:00
Jeltz	d16f444130	Use a dict for HBA hosts	2021-03-10 15:59:21 +01:00
Jeltz	4f6eda8329	Use /run instead of /var/run to please systemd	2021-03-10 15:57:19 +01:00
Jeltz	628e11488d	Switch postgresql to english	2021-03-10 15:22:01 +01:00
Jeltz	bd05b702bb	Use '::' in place of '[::]'	2021-03-10 15:19:39 +01:00
Jeltz	06b54d5f89	Use postgresql_privs	2021-03-10 14:27:14 +01:00
Jeltz	40eadf802c	Add template and no_log for postgresql_user	2021-03-10 13:58:40 +01:00
Jeltz	8e855d7009	Listen addresses must be quoted	2021-03-10 13:36:10 +01:00
Jeltz	7a07155237	Install python3-psycopg2 (required by Ansible)	2021-03-10 13:35:18 +01:00
Jeltz	36b04239fd	Rename 'postgresql_db' to 'postgresql_databases'	2021-03-10 13:34:58 +01:00
Jeltz	f919ec689a	Fix 'ansible_header' → 'ansible_managed'	2021-03-10 13:25:36 +01:00
Jeltz	9ef6202fdf	Add configuration for users and databases	2021-03-10 13:23:52 +01:00
pz2891	bbf4ac323c	Moniroting of ups environmental temperature	2021-03-10 12:55:11 +01:00
Otthorn	8b9bef865e	postgresql listen on pseudo-address	2021-03-10 12:26:18 +01:00
Otthorn	dbbaf0d26d	remove tailling whitespaces	2021-03-10 12:11:02 +01:00
Otthorn	a4c393d3fb	fix yaml ci truthy value	2021-03-10 12:10:06 +01:00
Otthorn	d14306a86c	fix syntax for CI	2021-03-10 12:08:05 +01:00
Otthorn	a625a58ddd	create role postgresql_server	2021-03-10 12:01:32 +01:00
Jeltz	2c0727a419	Update the list of packages installed via baseconfig	2021-03-07 21:41:42 +01:00
jeltz	41779fb172	Merge pull request 'Add backup root SSH keys' (#27) from add-ssh-keys into master ... Reviewed-on: Aurore/ansible#27	2021-03-07 21:30:38 +01:00
Jeltz	deb4372588	Merge branch 'master' into add-ssh-keys	2021-03-07 21:29:57 +01:00
Jeltz	929baa300f	Use 'update_motd' in 'prometheus_federate' (again)	2021-03-06 04:48:39 +01:00
Jeltz	71ee06c9c0	Fix typo	2021-03-06 04:45:00 +01:00
Jeltz	bc2701d8ba	Use 'update_motd' in 'prometheus_federate'	2021-03-06 04:43:09 +01:00
Jeltz	2353589da6	Ensures /etc/update-motd.d exists	2021-03-06 04:42:21 +01:00
Jeltz	1d0200a1f0	Use 'update_motd' in 'prometheus'	2021-03-06 04:32:06 +01:00
Jeltz	b81600aef8	Use 'update_motd' in 'baseconfig'	2021-03-06 04:31:20 +01:00
Jeltz	7e92fdfab7	Create an 'update_motd' role	2021-03-06 04:30:32 +01:00
Jeltz	cf07de4ec4	Fetch switch_snmp jobs	2021-03-06 02:41:58 +01:00
Jeltz	8abca7916f	Add switch_snmp job for prometheus	2021-03-06 01:57:32 +01:00
Jeltz	763cc2eb51	Generate targets_switch_snmp.json	2021-03-06 01:57:08 +01:00
Jeltz	eaa0d2e0fc	Fix bad indent in snmp.yml.j2	2021-03-06 01:56:18 +01:00
Jeltz	21fed6ae3f	Add useful lookups for switchs interfaces	2021-03-06 00:58:46 +01:00
Jeltz	52124d2cad	Cleanup prometheus_federate's prometheus.yml.j2	2021-03-06 00:46:13 +01:00
Jeltz	7d527be1c0	Remove duplicate alerts from 'prometheus-federate'	2021-03-06 00:45:43 +01:00
Jeltz	32669e1fb1	Don't load Django rules prometheus-federate	2021-03-06 00:44:22 +01:00
Jeltz	4ca7ebd144	Add a unique exported label (useful for federation)	2021-03-06 00:40:44 +01:00
Jeltz	802bfcc698	'prometheus-federate' must not retrieve its own federated metrics	2021-03-06 00:38:36 +01:00
Jeltz	958eaa1bcb	Use label federated_instance instead of instance	2021-03-05 00:54:44 +01:00
Jeltz	6525508401	Forward journald logs to rsyslog	2021-03-02 01:24:53 +01:00
Jeltz	77a5fdac6f	Remove some duplicate logs from syslog.log	2021-03-02 00:56:28 +01:00
Jeltz	529550f594	Don't use 'imjournal' ('imuxsock' is already used) ... I still don't understand why it increased the size of the firewall logs by a factor of 5 to 10, but we don't really need structured logs from systemd-journald and the author seems to discourage it's use, so I will not investigate further.	2021-03-02 00:46:16 +01:00
Jeltz	ee041b9ead	Use 'simple' instead of 'oneshot' (rotate service)	2021-03-02 00:14:25 +01:00
Jeltz	1f6bfeee23	Fix broadcast address on routeur-aurore	2021-03-01 20:04:38 +01:00
Jeltz	0f55b90de9	Remove 10.129.0.1 gateway on routeur-aurore-*	2021-03-01 20:04:02 +01:00
Jeltz	b13b22da05	Add ignored destinations for firewall logs	2021-03-01 19:39:11 +01:00
Jeltz	8f815a30c5	Remove useless date (already added by journald)	2021-03-01 17:47:12 +01:00
Jeltz	acd5721a5b	Fix typos in rotate-remote-logs.service.j2	2021-03-01 17:45:17 +01:00
Jeltz	9547868c7d	Send nginx logs to local syslog	2021-03-01 17:40:05 +01:00
Jeltz	cdb9f88614	Do not rate limit collection of journald logs	2021-03-01 16:31:52 +01:00
Jeltz	9eeb8ccd73	Remove non-Ansible SSH root keys	2021-03-01 16:08:08 +01:00
Jeltz	9252249d18	Use 'true' instead of 'yes'	2021-03-01 04:15:54 +01:00
Jeltz	e4b58c0bf4	Fix typo in 20-collector.conf.j2	2021-03-01 04:07:17 +01:00
Jeltz	c65b3f090b	Compress and delete old remote logs ... Logrotate is not used because I didn't found an easy way to configure it to handle the compression/deletion of log files already rotated by rsyslog (it is probably possible, but I found the script to be easier).	2021-03-01 03:58:58 +01:00
Jeltz	f7183095c1	Add explicit permissions for directories	2021-03-01 02:26:22 +01:00
Jeltz	ba8b4e8c29	Fix the ordering of rsyslog.d files ... A call to sendLogsToRemote for logs received through RELP/UDP has been added (to send them to Logstash/Redis/…), so common.conf's prefix must be lower than collector.conf's. Note: future "third-party" config files will also call sendLogsToRemote and thus will also have to use a prefix higher than 10.	2021-03-01 02:15:28 +01:00
Jeltz	7fd1b5ff5d	Add rsyslog_collector role	2021-03-01 01:27:56 +01:00
Jeltz	6263c31785	Add rsyslog_common role	2021-03-01 01:27:30 +01:00
Yohann D'ANELLO	ba6da939ab	[certbot] Fix certificates for auro.re ... Signed-off-by: Yohann D'ANELLO <ynerant@crans.org>	2021-02-24 13:57:59 +01:00
Yohann D'ANELLO	ae151321db	[nginx/certbot] Clone roles from Crans ... Signed-off-by: Yohann D'ANELLO <ynerant@crans.org>	2021-02-24 11:46:37 +01:00
pz2891	d7d0676f5e	Remove .save file; remove fo fleming prometheus	2021-02-18 17:53:15 +01:00
pz2891	74c30b81df	Merge branch 'master' into Global_monitoring	2021-02-17 19:41:06 +01:00
pz2891	b278b02bc2	Remove percentage sign for load alert	2021-02-17 19:37:33 +01:00
pz2891	0b90c9944b	Fix CI warning from last commit	2021-02-17 18:15:31 +01:00
pz2891	61001e09f5	Add alert for load usage	2021-02-17 18:08:39 +01:00
pz2891	a5b4deacee	Rename federate role; update of alerts of federate prometheus; update of configuration of federate prometheus	2021-02-17 17:42:24 +01:00
Solal Nathan	5b2580056d	🐛 Final fix, should stop sending ill-formed mail from now on	2021-02-16 18:10:39 +01:00