Ongoing server issues, machine switch tonight 23:00 (it never rains but it pours)
Tonight our VPS will be moved from one physical machine to another, staying with xtrahost.co.uk. This is an attempt to resolve the server's "AWOL" periods we have been experiencing. Despite extensive investigation neither we not xtrahost can find a reason for these problems, and we now want to eliminate hardware as a possible cause, hence the move.
Here's the plan. If anybody here in the IT group wants to make any observation, please do.
Last night we routed access to the site via an HTTP proxy elsewhere, meaning that we can quickly point everybody at the new server (new IP address) without DNS propagation lag, simply by pointing the proxy to the new IP (the proxy itself is at www.ksfiomdepositors.org, by now everybody accessing the site is going through the proxy.)
Copy of reply from xtrahost attached. In short, we're on for 23:00 tonight, new IPs 82.113.155.13 and 82.113.155.14 - I have set these up in DNS already as ip3.ksfiomdepositors.org and ip4...
Plan A
22:30 I will put the site into "maintenance mode" and take a backup of the database, just in case and for "Plan B" below.
23:00 Switch your proxy to point to 82.113.155.13.
23:30-00:00 The new server comes up, will still be in Drupal's "maintenance mode", but should be visible via your proxy as www...org as before. A few quick checks and then we switch back to on-line mode.
Once we confirm the server is working, I'll update DNS to point the www... there as well, so that over the subsequent 24 hours people should pick up the new address and so bypass the proxy.
Plan B
If after the server change things are no better (or even worse), what do we do? I suggest:
-
Set up on another server (your server?) to host the site temporarily. That's basically a matter of copying /var/www/drupal* across and importing the database. We could set this up as a safety measure during the time that xtrahost are doing the move. That way we know we can get back online one way or another fairly quickly.
-
Install a fresh OS on the new xtrahost server, probably Debian? Could be tonight or tomorrow. Another option is to switch to another provider, but why? xtrahost are not a bad company, just seems we have found a weird problem, and just maybe it's a broken OS install or something we (or I !!!) have done wrong.
IT - small problem with message times
I made a post at 9.30 local time (France) this morning (Friday). This was the time shown on my computer. In my book, as both UK and France are now on winter time, that means 8.30 GMT. The post showed up immediately on-line, but is dated "10.30 GMT". This is not a one-off - it's always been like that (including before the clocks went back last weekend).
Do others have a similar problem - or is it at my end? I checked my account settings - the time zone is correctly set as Europe/Paris. Can anyone in IT help? I know it's a minor problem, but would be glad if it could be sorted out.
And once again IT, you are doing a really really great job! Thanks for everything.
Tempus Fugit
Been like that a while I haven't bothered about it until saw the latest post here. It is now showing correct GMT.
Time display
Think I've tracked down the problem. It was wrongly putting (GMT) after what were in fact local times based on users' chosen time zones. If you are not logged in, display will be GMT (UTC). If you are logged in and have set a time zone, times will be displayed using that time zone.
timestamp.
I was having the same problem, and I switched from London to UTC, and all is well. A quick search of google indicates this is a known "feature" od drupal and patches are available. I think selecting UTC for those who are concerned is a practical workaround
Drupal time bug
Thanks for that bit of research - I'd come to the conclusion from the comments so far that it must be a Drupal bug since everything seems set up correctly. I'm kind of reluctant to install patches unless absolutely necessary (potential maintenance problems when upgrades come out) but will take a look at what drupal.org are saying about the topic.
Time errors
Will try to sort it - a quick look indicates that all is well (which it obviously isn't, from you report.) As I write this it's 23:58 which should be 22:58 GMT, I'll see what the time-stamp is.
Time-stamp
Seems correct (an extra minute whilst I was typing). This should get stamped as 23:01 GMT.
Time stamp
NG seems the time stamp is affected by whatever we have set up in our accounts - reflects relative local time setting rather than GMT.
Your mesage at 23:01 was showing 03:01 on my machine with the time zone set to Asia/Muscat. When i set it to Europe/London, your message then displayed with the correct time stamp of 23:01 - guess the stamp should be dependant on the poster's computer and not the reader. Hope this helps
time stamp - solution
Yes - that seems to be the case. I've changed my setting to London time (tho' it says I should select my local time) and that seems to work. Thanks expatvictim.
ng - Time-stamp
Thanks for trying ng. But with me it says Sat 01.01. So somehow it seems as though the 1 hour time difference gets added instead of being subtracted. I note that some new posts do appear to have the correct time - maybe these are those originating in UK? Where are you based? If the problem is at my end, any idea how I can correct it?
ng- again
and my reply above got stamped 10.58, when it was 9.58 here, so should be 8.58 GMT ....
You IT guys are fantastic - how can we help?
Your work and the whole site is SO APPRECIATED. Many many thanks. Are you funding this from your own pockets? How can we donate something to your expenses. Can some sort of 'Contribute/Pledge Here' link be set up under 'Important Please See' so that we can offer something via PayPal, e-transfer or whatever. I'd like something to go also to the London/IoM teams, etc. What do you think?
Trying to sort out funding
Thanks... we will post on the site once we've figured out how to handle funding. The truth is we do need some.
An update
Right now the site is still on Stef's VPS. Meanwhile xtrahost.co.uk have set up our new VPS there and from testing it appears that the "AWOL" problem is gone.
So now it only remains for us to move back to the xtrahost server. This will be a relatively simple operation, as the infrastructure is already in place - we just need to shut down for a while, copy the live database across and then update DNS. We will still have the proxy in place to provide access while DNS is propagating. The whole thing should be no more than half an hour of downtime, and hopefully 15 minutes.
So, just waiting for Stef and I to find a mutually convenient time to do this, but it will be in the next day or two.
Our Heroes
To all those putting in the hours, their hearts, money, sweat, blood and tears -- MatWalker, changamire, Diver, expat, ng, Sleepless, Teapot, Jenren, occams razor, Lucky Jim and those I should have mentioned:
THANK YOU!
We will succeed, because we are a highly motivated group of people prepared to work as a team. An examplary demonstration of the power of the web.
NG -
Thank you so very much for all your hard work ..... you don't know how much comfort this website gives to so many of us on a daily basis.
I agree this website does a fantastic job -
I agree this website does a fantastic job and is a great comfort - has certainly become top of my bookmarks! All of my top seven are now connected with this mess - with HM Treasury being the bottom of the seven as being the least useful of the crop!
To everyone in London, IOM, the website location and around the world, I will say well done and keep it up.
I know many, many are affected on a daily basis much more than myself, but rest assured that those suffering can be felt at whatever distance.
I am quietly confident we will proceed to victory somehow.
BF
Thank you, daughter
The web team are not as high profile as the London and IoM teams, but we are an active part of the Group, as indeed is everybody who contributes here.
Stef's help has been invaluable, and we have ongoing important though relatively unnoticeable contributions from our Site Editors - occams razor, Lucky Jim and others. And let's not forget MatWalker and changamire who originally set up the site and so without whom perhaps none of us would even be here.
Sometimes I get less than positive moments when I wonder if we will succeed. Then I remember, we are around one and a half thousand people united by a common need, communicating and ever increasingly organised and focussed. How can we possibly fail?
well done
Thank you for all you are doing on this site.
As a complete computer dumbo - showing my age I'm afraid- I am stunned with admiration for those who have control of these things.
Masters of the Universe indeed!
quite agree, ng and his/her
quite agree, ng and his/her pals are brilliant.
Thanks! For the record, I'm a
Thanks! For the record, I'm a bloke – the name is rather ambiguous, sorry :)
Nice one guys!
Not sure most folk on the site realise just how much work goes into keeping a system like this running sweetly! You inherited a bag a nails and massaged it into a bl**dy good site.
You can always tell 'good' techs - they plan for disaster while keeping fingers crossed all will be well!
A Tester.
Plan B is in operation
In fact, things were so bad today that we went straight to plan B. Kudos to stef for taking the decision and pulling it off all single handed (I was away with other commitments.)
So, now we are going to start afresh on xtrahost with a clean install and move back. This will happen over the next 24 hours, though no great rush as currently things are working ok.
Traceroute
Traceroute sometimes helps to locate network problems. How to: http://www.more.net/technical/pingtrace.html#tracert
www.ksfiomdepositors.org currently traces to thor.stefpause.com but earlier today traced to ksfiomdepositors.netgenius.co.uk which seems a bit strange if you are currently with xtrahost.co.uk
FWIW, testing http://www.ksfiomdepositors.org/ at http://www.rexswain.com/httpview.html shows up a Warning data·error·in /includes/bootstrap.inc though I don't know what that might be.
From my limited knowledge, I would suggest a DNS issue, but don't really know. I agree with bypass the proxy. Ultimately I think you want a working server with www.ksfiomdepositors.org pointing to that server's nameservers, and maybe the previous domains ksfiomdepositors.netgenius.co.uk and www.ksfiomdepositors.com "301" permanently redirected to www.ksfiomdepositors.org (if that makes sense).
Apologies if you've gone through this already.
Technical problem - internet site blocking
Just wanted to mention that as of around 1300 CET this afternoon, this site became blocked from my office PC as it was designated as inappropriate (social networking, inappropriate language or content etc). Up until then I had no problems with access.
Not sure if something changed in the configuration at that time but I'll keep my fingers crossed that I can access it again tomorrow after this server change.
What I have gained in productivity I have lost in peace of mind. Thanks for all the amazing work you're doing on the technical side.
Still blocked?
Hi again Swissloss, is the site still blocked at work for you? If so, would it be possible for you to ask the IT people at your office what site filtering software they're using? (E.g. Checkpoint's Surfcontrol, SmartFilter, etc.) There are few things I may be able to do to help (ICRA codes for the site, checking if we're listed in certain filter lists, etc.). Ta!
Stef - site still blocked
Hi Stef - thanks for checking - wish you were my IT support person ;0)
Yep site is still blocked but I will see if I can check with IT in a roundabout way about what they're using.
Keep up the fantastic job!
It's possibly down to my VPS
It's possibly down to my VPS provider's netblock being blacklisted at Spamhaus. I notified them as soon as I noticed, so hopefully this'll be resolved in the next couple of hours. It shouldn't matter too much, anyway, as we'll be moving the site again – hopefully for the final time!! – at around 2300 UTC this evening. (And the new IP addresses aren't blacklisted.)
Thanks for that, P.Taylor,
Thanks for that, P.Taylor, we're on the case. It's not a DNS issue, it's our Xtrahost VPS playing up.
Thor's my VPS - I migrated the entire site temporarily an hour or so ago to my server as the original server's having some very strange issues. We're working with the host to try and solve the ongoing problems with the original VPS, but in the mean time we had set up my server to act as a reverse proxy and consolidate the large amount of differing domains and redirects into one place. The proxy errors displayed only occurred when the original server was down - the proxy obviously can't forward you to a site if the site in question isn't responding.
There are still some lingering issues with Drupal due to me migrating it - I think it needs its caches clearing but I don't have admin access to the actual software, so it'll have to wait till NG gets back in.
Also - site e-mail may well not be working at the moment, due to it being on my site.
Sorry again for all the hassle, folks. Believe me, it's been frustrating for us, too!
Wrong Time
Only a trivial matter, I noticed the site clock is or maybe by now was, two hours ahead of real time. ? somewhere in Eastern Europe! my post http://www.ksfiomdepositors.org/node/982#comment-12823 was definitely made at 22:29 on 29 October not 00:29 on 30 October!
Domain DNS check reports critical errors
Stef, it may not be a DNS issue, but the Domain Health Check at http://www.dnsstuff.com/ says "You have (2) critical errors which require a fix." It may be worth checking what those errors are?
Unfortunately DNSStuff
Unfortunately DNSStuff reports aren't free any more (they used to be – well, ad-supported, anyway) so I can't check what they think's wrong :)
I'm pretty sure I know what the issues are, though, and they're all minor: the TTL value's a little low (no biggie, and actually a good thing for us at the moment); reverse DNS doesn't match (only matters for mailservers); IP is listed in the Spamhaus SBL (again, a mailserver issue). As the site's getting moved to the new server at 2300 UTC, it should be OK till then.
Check out
Check out http://www.intodns.com/ it does pretty much everything DNS Report used to but for free :o)
There's nothing wrong according to that though. Shout if you need any DNS help, I've got an account with a DNS provider where I host the DNS for a whole bunch of domains.
Ben
Ooh, thanks for that, Ben,
Ooh, thanks for that, Ben, didn't know about IntoDNS. Ta!
Great stuff - server switch
Good luck with the switch. You're doing a great job.
P.
Thanks! I'm almost looking
Thanks! I'm almost looking forward to the switch, just 'cause I won't have to spend a couple of hours every day on backend webserver admin – I'm sure Andy feels the same :)
As i do not speak your IT
As i do not speak your IT language, I'm just delighted that there are people out there who do. These days I'm just a Mac user who just "uses", and all this other stuff has progressed so far and so fast over the last 10 years I simply don't understand anything any more!
Blessings on you and ng for what you're doing.
By the way, whatever this is, it works better/faster than anything I've seen since it started!!
And I repeat many earlier comments about funding whatever is needed.