Discussion
*Sigh*
OK, niche.
I don't understand enough about this to be able to take succinct advice or suggestions, and I don't know enough to be able to provide all the information that people will need to be able to help me.
Kind of a Catch-22 situation.
However ... I have a problem with #Email ... feel free to read no further.
Your chart is ready, and can be found here:
https://www.solipsys.co.uk/Chartodon/117569958998366993.svg
Things may have changed since I started compiling that, and some things may have been inaccessible.
In particular, the very nature of the fediverse means some toots may never have made it to my instance, in which case I can't see them, and can't include them.
The chart will eventually be deleted, so if you'd like to keep it, make sure you download a copy.
Sending email via my usual webmail interface is working fine.
Sending email via the command line utility that I use is *mostly* working fine.
There is one destination (that I know of!) that is rejecting email with:
550 5.7.1 Message rejected:
Message lacks SPF/DKIM/DMARC Auth Records
So they get delivered via the web interface, but not via the script I'm using.
* I own my own domain
* I can connect via shell
* it is shared hosting
* I do not administer the machine,
* My access to logs is limited
* my options are limited.
I'm currently *really* busy with other things, but it would be really nice to get this fixed. If someone can provide me with the right level of "This is what's wrong" then I'd be very grateful.
The script I'm using to send email is here:
https://www.solipsys.co.uk/EmailAnon/Mailer_anon.txt
That's a Python2 script ... it's not good code, it's very old, it's not stripped to be minimal
My next debugging task would be to make a minimal version that shows the current behaviour, but I don't have a lot of time right now.
I'm asking in case this is an easy fix ... and if not, to save the information for when I get time.
And thank you everyone who has replied so far.
I will continue to add to this thread as and when I can.
One friend says that a recent email from my domain passes SPF and DKIM but not DMARC.
This thread seems also to point the finger at DMARC:
https://social.mythic-beasts.com/@beasts/117569497236204746
(Information provided in other posts in this sprawling thread)
@ColinTheMathmo indeed DMARC seems to be missing on your domain, so that might be the problem, if the recipient is one of the providers that require DMARC.
Tested here: https://mxtoolbox.com/SuperTool.aspx?action=dmarc%3asolipsys.co.uk&run=toolpage
Also from the command line: "dig _dmarc.solipsys.co.uk" doesnt return a suitable header.
If you have control over your DNS then adding a basic "don't care" DMARC entry as per that post might work. Look for a guide for adding DMARC TXT entries for which platform hosts your DNS.
@ColinTheMathmo your SMTP server should be adding the relevant headers for DKIM. Are you in control of the SMTP server?
@ColinTheMathmo Others are having the same problem, see this thread: https://social.mythic-beasts.com/@beasts/117569497236204746
Seems to have information about solutions too, but I've not gone through the detail.
@semiprime That all looks relevant ... I will pass it on to the person who administers my machine.
Thank you.
@ColinTheMathmo @semiprime looking through that, it seems that your DMARC might be the problem.
I don't think (running `dig _dmarc.solipsys.co.uk TXT`) that you've got a DMARC record in your DNS.
From https://know.me.uk/@CenturyAvocado/117569556214827513 it might just need a TXT record adding to your DNS for _dmarc.solipsys.co.uk - seems it might just need to contain "v=DMARC1; p=none;"
(`dig _dmarc.doesliverpool.com TXT` will give you a comparison showing ours, but I don't think you *need* the rua/ruf stuff)
@ColinTheMathmo I do administer my own mail server, and have set up SPF/DKIM/DMARC, so may be able to help (although it’s been a while so might need to refer to my notes).
It’s a shame the error doesn’t say which of the 3 technologies has failed.
For a primer of what they are see https://www.cloudflare.com/learning/email-security/dmarc-dkim-spf/
My understanding is SPF & DMARC restrict which server can send email for your domain, and DKIM cryptographically signs the email to prove it came from that server.
(continued below…)
@ColinTheMathmo so either the script is sending from the wrong IP address (which would fail SPF or DMARC) or the email isn’t getting signed by DKIM.
Your webmail might be configured to send via an SMTP on a different server, which would have the correct IP address and DKIM encryption, whereas the script might be using a native SMTP server on the same machine. Worth checking the webmail config.
(continued below…)
@ColinTheMathmo Or the webmail might use SMTP on the native machine, but the script bypasses that SMTP service, either using an external SMTP server, or talking SMTP directly to the recipients server.
If you can give more details of the script (what language: PHP, Perl, shell, etc) people might be able to give advice on how it sends its mail.
@ColinTheMathmo AIUI the SPF record (which is a DNS TXT record) lists the hosts that are allowed to send email, so you might need to add the machine-running-your-CLI to it?
Maybe the host sending things from webmail is listed in there but not the CLI machine?
@amcewen The script connects to the regular email host and sends from there:
server = smtplib.SMTP(server_host_name,587)
So the emails should not advertise themselves as coming from my local machine where the script is running.
@ColinTheMathmo Is that 550 error from a Microsoft domain by any chance?
@penguin42 From hotmail ... yes.
@ColinTheMathmo @beasts was asking about this today - they say they saw a big increase of problems from MS with apparently correct DMARC.
@penguin42 @ColinTheMathmo You need to add a DMARC record. Even just p=none should do the trick. MS seem to have had a significant unannounced policy change yesterday morning. https://status.mythic-beasts.net.uk/
@beasts That's certainly looking like "A Thing", though there may be other problems.
I'll get onto my site admin to get them to do that ... thank you.
CC: @penguin42
@beasts Even so, it doesn't explain why email from the webmail client gets through.
So I'm confused ... but I believe that's the normal state of affairs when dealing with email systems.
CC: @penguin42
@ColinTheMathmo @penguin42 Feel free to send test mails via both routes to mail-tester.com and then mail a link to the results to support@mythic-beasts.com (FAO pdw). I'm keen to better understand what's going on here.
@beasts I've created a DMARC entry now, so maybe it's fixed.
Even so, I will do this for you in the next hour or so.
CC: @penguin42
@ColinTheMathmo what the message "Message lacks SPF/DKIM/DMARC Auth Records" means is that the domain of the email address that the message claims to be from doesn't advertise those records.
If you want to send an email from IP address 1.2.3.4 claiming to be from test@example.com, then the example.com domain needs to have an SPF record saying "1.2.3.4 is allowed to send email claiming to be me", and if that is missing then recipients will treat the email with suspicion.
@ColinTheMathmo My first guess - given that everything else works - would be that your command-line sending script is not setting the "from" address correctly. Recipients that don't check SPF etc don't care who an email says its from, but recipients that do check will reject it.
@bencurthoys The "From" field seems to be being set correctly, from what I can tell.
If you DM me an email address then I can send test emails to you from both the webmail and my script and you can see if there's a difference.
That would be an instant and easy fix.
@ColinTheMathmo obvious question is which command line utility.
@Colman Script is here:
https://www.solipsys.co.uk/EmailAnon/Mailer_anon.txt
Yes, sending via webmail to the same destination works.
