Over the past month or so, we’ve noticed a considerable uptick in issues sending mail to Gmail or other Google handled email addresses. Messages that had been going out to Google’s mail servers were now starting to fail. This is a perfect example of how systems on the Internet are always evolving. We receive a lot of statements from users saying “Well, these messages used to work, I haven’t changed anything.” This may very well be true – but you have to understand that all of the framework and systems that control the Internet are always changing. Just because something used to work on the Internet, doesn’t mean it will always work.

From what we have been able to ascertain about the changes Google has made, they have increased their grading on SPF records. SPF is a special DNS TXT record that a domain owner uses to identify what IP addresses should be sending out mail from that domain. It’s an email authenticity check so that receiving mail servers (i.e. Google’s mail servers) can determine if mails from the domain and from that sending server IP address are legitimate.

If you have not already managed an SPF record for your domain name, you can access this in your cPanel by clicking the Email Deliverability link and then clicking the Manage button next to the respective domain name.

If you do not see the Email Deliverability link or if you need help with this, submit a support ticket and our support team will be glad to help you with this.

Email Forwarders

The other issue that this Google deliverability and SPF requirement involves concerns the use of Email Forwarders.

We have for a long, long time strongly discouraged users from setting up email forwarders that forward mail to an external mail server (i.e. Google’s mail servers). If you log into your cPanel and click the Forwarders link, you will see a notice that states that off-site email forwarders are discouraged.

Unfortunately we can’t stop users from setting up these forwarders, but you should understand that if you set up such a forwarder you may encounter issues with email deliverability. Additionally this email deliverability issue can lead to affecting other users on the server.

Email forwarders to an off-site or third-party mail server are just a bad, bad idea. Remember, the systems that control the Internet’s infrastructure is always changing. When SPF first became a thing with identifying email legitimacy, concerns with external mail forwarders was a noted issue. But the developers of SPF had to weigh the usefulness of SPF in identifying email authenticity (which was becoming more and more of an issue) against the use of email forwarders (which was largely in decline, due to smartphones and largely an always connected world).

You should understand that because of the way SPF and forwarding works – if you have any email addresses that forward off of the server, then all mail sent to that address and forwarded on to it’s destination is ALWAYS going to fail SPF.

With Google taking such a hard-nosed stance in regards to SPF authenticity, that means email addresses on our servers that are forwarding to Google mail servers, are always failing SPF. As these failures build and build, Google starts to identify our servers as a possible spam source. This is why forwarding mail off of the server is such a bad idea, and it’s why we have been discouraging it for several years now.

If you have any email addresses that forward off of the server – and I should mention that this includes any off-site server, Hotmail/Outlook/Microsoft, Yahoo, Google, AOL, etc, not just Google – we would really, really encourage you to remove the forwarder and set the email address up as a real email account on the server and check it directly. This is going to benefit you in terms of receiving all of the mail sent to the email address and it’s going to benefit us by lessening the SPF failures sent by our servers that hurts our server’s email deliverability reputation.

You can setup a real email account for the email address by using the Email Accounts link in your cPanel.

Shortly after our previous post, Ioncube released version 11 of their encoder and loader. This is a major release from Ioncube. However, this major release does not include support for PHP 8.0. This speaks volumes to Ioncube’s commitment to staying current with PHP development. We no longer expect PHP 8 support in Ioncube to be any where near imminent.

After this announcement we decided to take a look and see just how many PHP scripts on our servers are using Ioncube encoded files. This is honestly something we should have done many months ago – but again we though PHP 8 support in Ioncube was forthcoming. The results of our findings was “not many” Ioncube encoded scripts exist on our servers.

Given all of this, we’re going to proceed forward with slowly switching accounts over to PHP 8 by default. Effectively, this means that if you’re account is using PHP 7.3 it will be switched to PHP 8.0. We want to introduce this slowly because we’re unsure of how this will affect real world application use. If we start to receive reports of issues, then we will tap the breaks on this.

How can you prepare yourself for this?

The best thing you can do is to make sure your scripts are being kept up to date and that any plugins, themes, components, or extensions are also being maintained by reputable developers and being kept up to date.

One of the more popular scripts/applications used on our servers is WordPress. We’re told that WordPress 5.6 and higher should be compatible with PHP 8.0. WordPress 5.8 is currently the latest branch to be developed and is more likely to be developed with PHP 8.0 in mind. If possible, we would encourage you to insure that your WordPress script is updated to WordPress 5.8.

WordPress has a very extensive plugin and theme system. Keeping these up to date and insuring that they are being maintained by reputable developers. However, it’s just impossible for us to know what plugins and themes are still being maintained and are compatible with PHP 8.0. The best advice here is to contact the developer of the plugin or theme in question. Any PHP developer that’s worth their weight will be aware of PHP 8.0 and PHP 7.3’s end-of-life. If they’re not – then that tells you the plugin or theme probably isn’t that reputable. If you contact them and don’t get a response then that would tell you that the project has been abandoned.

Other scripts and systems that aren’t associated with WordPress would also need to be properly updated and maintained.

If you do encounter an issue after being switched to PHP 8.0 – then that tells you that you have some issues that need to be addressed. Either the script, plugin, theme, component, extension, etc is not up to date or is not being properly maintained.

If you do encounter issues after being switched to PHP 8.0 – we can downgrade you back down. We’re not cutting PHP 7.3 from our systems entirely just yet. But we do want to raise awareness that if your script or application won’t function under PHP 8.0 then you’ve got some issues with that script or application that needs to be addressed.

Edit – November 28, 2021 – there is updated information about this in our new post.

Some of you may be seeing warning or notices in your web scripts or applications about the upcoming end of life for PHP 7.3. These concerns are true and valid. Official support for PHP 7.3 will end on December 6, 2021. This does not mean that PHP 7.3 will stop working on December 6, 2021 – it simply means that the PHP developers will no longer release any security fixes for the 7.3 branch of PHP.

All accounts on our servers are using PHP 7.3 by default. The term “by default” means that unless you have made arrangements with us to change the PHP version – then your account is using PHP 7.3.

We prefer to hold true to these development environments as much as possible. That means we would prefer to move off of PHP 7.3 by default before the December 6, 2021 cutoff. But we haven’t yet done that. Why?

The majority of the reason revolves around Ioncube. Ioncube is a very popular PHP encoding system – think of it as a way to distribute code without exposing the exact source code for that project. As of now, Ioncube still hasn’t released updated encoders and loaders for PHP 8.0. Why? We have no idea. Ioncube is severely dragging their feet in resolving this. As a leading encoder for PHP, Ioncube should have been working on a PHP 8.0 encoder/loader well before PHP 8 was release to the general public. The first alpha of PHP 8 was released on June 25, 2020. And Ioncube still has no answer for PHP 8.

Ideally we expected Ioncube to release an encoder/loader in late Summer 2021 at the latest. We had plans to begin shifting PHP by default to PHP 8.0 around the first of October 2021. But Ioncube never released an encoder/loader and now we’re knocking on the door to December and the December 6, 2021 deadline for PHP 7.3.

What about PHP 7.4?

PHP 7.4 is also out there and in life (until November 28, 2022 – so about a year from now) and an Ioncube encoder/loader is available for PHP 7.4. Typically we like to skip versions when setting a default PHP version, otherwise we’re changing default PHP versions once a year or less. We did this with the PHP 7.1 to PHP 7.3 release cycle, we skipped over PHP 7.2 as a default version. This was our intended target for PHP 8.0 – go from PHP 7.3 to PHP 8.0, skipping PHP 7.4.

Believe me when I say that Ioncube dragging their feet on this has been a huge obstacle for us, because we really did not plan for Ioncube to take this long to release an encoder/loader for PHP 8.0.

What are we going to do now?

We don’t really know. We’re playing this by ear now. From the looks of things, unless something unexpected happens with Ioncube and PHP 8.0 within the next couple of weeks, we’ll probably continue with PHP 7.3 by default after the December 6, 2021 end-of-life and re-evaluate all of this about a month later at the first of January 2022. At that time we’ll probably switch to either a PHP 7.4 with Ioncube by default or a PHP 8.0 without Ioncube by default.

Continuing to use PHP 7.3 after it’s end-of-life isn’t the end of the world. The chances of a security concern to arise in PHP 7.3 soon after it’s end-of-life is very small. However it is not a good idea to continue using PHP 7.3 long term after it’s end-of-life. We’re aware of this. But we were just really hoping to have PHP 8.0 with Ioncube available by now.

How do I know if I am using Ioncube?

If you purchased any software to use on your website then it’s a good bet that your PHP scripts are encoded with some encoder. Ioncube is not the only PHP encoder, but it is the most popular one. Just how many of our clients are using Ioncube encoded software will factor into our decision to my to PHP 7.4 with Ioncube vs. PHP 8.0 without Ioncube as the default option.

If you are using a piece of software that is encoded with Ioncube I would strongly encourage you to discuss this with that software’s developer and ask them if it’s possible for them to move the encoding to a different encoder – one that supports PHP 8.0.

This is not the first time Ioncube has dragged their feet on this. When PHP 7.0 came out they were just as slow at releasing an encoder/loader for PHP 7.0 as they are now.

I understand that users may be disgruntled with this. Believe me, we are too. But I do not know of anyway to get Ioncube to speed up their development and release process.

Edit (Wednesday, September 8th, 2021) – to include WinSCP as the preferred SFTP client.

We are planning to disable FTP on all of our servers at some point in the future. But that doesn’t mean you won’t be able to use FTP to manage the files on your website.

Instead of FTP, we will be replacing the service with SFTP. SFTP functions practically identical to FTP, except it’s simpler and built from a secure foundation.

Most modern FTP clients will function as an SFTP client as well. Modern application developers have known about the shortcomings of FTP for years and have been preparing for its demise. Unfortunately there are just too many FTP client applications out there for us to provide a guide for each and everyone of them. The bottom line is, you need to check with your FTP application, probably in the Site Manager or Connection Manager of the FTP application, and check to see if it supports the SFTP protocol for your web hosting FTP connection. You simply need to change the protocol to use SFTP and specify an SFTP port – which for our servers is Port 9122.

We’ve taken to recommending WinSCP as the preferred SFTP client. You can download WinSCP free at:

https://winscp.net

Below we have some images from Winscp’s Site Manager to illustrate these changes.

The key takeaways here:

File Protocol: SFTP
Hostname: %your_domain_name%
Port: 9122
Username: %your_webhosting_account_username%
Password: %your_webhosting_account_password%

And that’s it. Now when you connect, you’ll be using SFTP and connecting securely.

Why are we making this change?
FTP by nature is not a secure protocol. That means everything you send along the connection is not encrypted and is viewable to anyone that might be listening on the connection. While the connection can be made secure, it’s just difficult to enforce this because FTP wasn’t written with security like this in mind.

FTP is also a bit more complex. It consists of a control channel and a data channel. This made more sense back when Internet connections were not as robust as they are today. It’s just a sign of times, older technologies always get pushed out and that’s whats happening with FTP. But again, SFTP is largely a drop in replacement that only requires a slight configuration change by the end-user.

Additionally cPanel is moving to dropping FTP on their platform, so a move to rid ourselves of a reliance on FTP is needed.

When will FTP be remove?
We don’t have an exact date. We’re waiting to see how well our users embrace SFTP and how quickly they can move off of FTP. We’re probably targeting late 2021, although this could also change if cPanel moves up it’s deprecation of FTP.

We will be sending out notices to users on our server that still use FTP and notifying them of this upcoming change. The hope is that all of those users will be able to seamlessly move to SFTP and this change can move forward fairly quickly. However, if users are relying on very old FTP applications that may make the shutdown more difficult. The concern with users that are using very old FTP applications that do not support SFTP is that they may be leaving themselves vulnerable to other security implications by relying on very old and antiquated software.

There’s been a sharp uptick in malware, keylogging, and ransomware attacks over the past several months. The move to disable FTP and replace it with SFTP is meant to better safeguard our services from these attacks. In security you’re only as secure as your least secure entry point and right now, FTP is one of, if not the, least secure entry points on our servers. So removing FTP is going to serve to better secure our servers and your account.

Let’s start out by addressing one fact – The Internet is ever evolving – is this a fact that we can all get behind?

To put in a different perspective, is the Internet you see, access, and interact with today the same Internet you saw, accessed, and interacted with back in 1995?

For me, the answer is a clear No. In 1995 we didn’t fiber Internet connections. We didn’t have dynamic and interactive websites. We didn’t have streaming TV platforms. We didn’t have Facebook and Twitter and other social media platforms. We didn’t have smartphones and Internet and communication in our pockets, always connected. The Internet has changed.

Despite all of this – Email has remained largely the same. It is still a very popular communication method, although instant messaging and text messaging have carved a niche in easy and simple communication purposes.

While the Email protocol hasn’t really changed – the abuse to email has. SPAM is rampant with email – I think we can all agree on this – and there’s no end in sight. As a result of this, a lot of the major email providers Microsoft, Google, Yahoo, Verizon, etc have really clamped down on their anti-spam and anti-malware methods. It’s becoming harder and harder to get legitimate mail through due to these measures.

This has a direct correlation to email forwarding. In the ’90s and early 2000s forwarding mail may have been common place. But as more and more SPAM reached these major email providers, the notion of forwarding mail off of the server became more and more problematic.

When you forward mail off of the server – when it reaches it’s final destination, that provider (typically Microsoft, Google, Yahoo, Verizon, etc) sees the message as being sent from our server. If it turns out that that message is a SPAM message guess what server gets deemed as the source of that SPAM? Our server. And when your server gets flagged as a SPAM source, nobody gets any messages from our server with these email service providers.

In this scenario our server becomes a “man-in-the-middle” Instead of the message going directly to the intended recipient it routes through our server first. This additional step also creates additional problem points. The simpliest path is usually the best path. Sending a message directly to the intended recipient is the best way to insure deliverability.

This is why forwarding mail off of the server is a very, very bad idea. In fact, that’s why we have an advisory posted in your control panel about such actions.

Over the past several months and years we have seen a huge increase in the amount of problems our users and servers have had in sending mail to a lot of these major email service providers. We do not believe these issues to be tied to direct SPAM or abuse on our servers (these issues usually show themselves with being tied to popular public blacklists and our servers remain clean on these). Instead we believe the issue is related directly to users forwarding mail off of the server to these various major email service providers. SPAM being sent to these forwarding email addresses is resulting our servers being blocked and blacklisted by these major email service providers.

What is the solution?

The best solution is to setup all of your @yourdomain.com email addresses as real email accounts and check them directly or via webmail. Alternatively, some major email providers provide an option to download messages via POP3 into their service. I know Gmail has such a feature – in your Gmail account go to Settings -> Accounts and Filters -> Check mail from other accounts. Other service providers may offer this, you would just have to check with them. If you’re only concerned with checking your Gmail or other major email service provider then this may be an option for you.

One of the common refrains regarding this is – “But, I’ve always forwarded mail like this and it’s worked” – but again, this goes back to what we first talked about at the opening of this post – The Internet is ever evolving and changing. Just because some activity used to work, doesn’t mean it will continue to work that way forever. A blocky, scrolling text marquee based website that was appealing in the 1990s isn’t going to be appealing to the generation visiting that website today.

The bottom line is – forwarding mail off of the server really needs to stop. It just doesn’t work in the Internet of the 2020s. You can apply different band-aids to try and get around it and temporarily fix it, but it still doesn’t change that it’s a horrible idea.