Behind the scenes tonight in the "IT Dept"

  • ng
  • 11/10/08 31/12/20
  • a depositor
  • Offline
Posted: Thu, 06/11/2008 - 22:08

The server problems today (similar to those of a week or so ago) have us mystified.. But, to show we're not twiddling our thumbs, and for those who are interested in these things, here is the Skype conversation tonight with associated teamwork, which finally got the server back up and working properly (fingers crossed) at 21:10 UK time - the times shown here are 1 hour ahead of UK ....


19:41:17] ng: Stef, how do you normally charge for your time? hourly rate, package price? Let me know your fees. It's getting to the point where we need to start charging the group for our work... this is not a hobby, and whilst I have a vested interest.. its only up to a point, if I'm not careful I'll lose more through lost earnings than I would lose with my deposit.


[20:55:05] Stef: Hey fella. Usually around £40 an hour for tech consultancy – should probably be close to 10x that but I've always been rubbish at working out rates!
[20:55:37] … The site's going horribly wrong at the moment, the usual issue - load averages of a ridiculous 25
[20:57:28] … 500MB+ swap, but I'm not at all sure what the hell's using it. Have restarted Apache and MySQL, but it's not helped. We need to wipe it all out and start with a fresh VPS, unfortunately. Ah well!
[20:58:10] … I'm tied up tonight and most of tomorrow, may be able to sort something on Friday night or Saturday?
[21:00:34] … Am rebooting the server now, hope that's OK - can't really hurt as it's already practically down :(
[21:05:21] … Recommend the following: Point the DNS back at my VPS now. Migrate back to my VPS in a day or two. Kill the VPS, get a fresh one & set it up, migrate the configs, etc. Migrate back. Cross fingers and hope whatever the hell's been going on doesn't happen again!


[21:20:50] ng: Hi Stef,
[21:21:31] Stef: Hey :)
[21:22:02] ng: Re. rates, yeah 30, 40, 50 ,60, 70, 80 ... all depends
[21:22:14] … I was even doing some work at 25, but that was for a charity
[21:22:45] … also agree. restart does not help... it does not seem to be a memory leak or runawar processes
[21:23:08] Stef: For this 25's fine, to be honest, as it's pretty much a charity/non-profit.
[21:23:30] … Load averages are still mental after the reboot, in the 25-30's
[21:23:56] ng: trying to work out some kind of remuneration... cant go on forever like this.. no guarantees but lets see
[21:24:21] … anyway, back to the server... so what happened? four or five days absolutely perfect... then wham!
[21:24:28] … doesnt make sense
[21:24:29] Stef: Aye, know what you're saying and agree, must be a real drag on your time :(
[21:24:56] … Yeah, it's WEIRD. vmstat doesn't show anything out of the ordinary. The only think I can think of is that the server's been rootkitted and someone's slamming the f*** out of it
[21:25:20] … But we can't see it 'cause all the tools are patched to hide it.
[21:25:25] ng: you mean the VPS or the PM ?
[21:25:30] Stef: the VPS
[21:25:34] ng: seems unlikely either way
[21:25:38] Stef: But it doesn't really make sense
[21:25:47] … Yeah, it's really unlikely
[21:26:33] ng: there is a difference this time to last time.... last time couldnt see any processes... this time see mySQL using lots of CPU, alos evident on the mysql admin graphs
[21:26:35] Stef: I did do a load of checks like rkhunter and chkrootkit and they didn't find anything. Dunno, it's pretty baffling.
[21:26:45] ng: processes hit 30 or 40... normally they run at4 or 5
[21:27:00] Stef: I saw that on the old server via mysqladmin (tunnelled)
[21:27:21] … (Same VPS on the old server-host.)
[21:27:32] ng: did you, ok, i never saw it... but it seems mySQl is the problem... or maybe it is effect rather than cause
[21:28:17] Stef: Nah, it's not the problem, I don't think. Like you say it's just an effect of the server being weird – makes MySQL go weird with it.
[21:28:32] … Anyway, we need a plan...
[21:28:47] … Can you change the DNS and point it back at my server?
[21:29:21] ng: yeah but it will take time
[21:29:27] … i mean delay...
[21:29:33] Stef: I know :(
[21:29:43] … Nothing we can do about that.
[21:30:07] ng: but think.. where the hell is the problem?
[21:30:29] … it has been fine, and nothing has change (well.. the database keeps getting bigger... maybe relevant)
[21:30:34] Stef: I dunno. The VPS itself is the logical conclusion.
[21:30:44] ng: it worked fine on yours
[21:31:00] … and on mine... apart from the outages which were not related
[21:31:17] Stef: Go in, stop Apache and MySQL. Check the load averages
[21:31:18] ng: but these guys do seem to know what theyre doing
[21:31:21] Stef: And the swap.
[21:31:28] … Bet they'll still be mental.
[21:31:42] ng: even with apache stopped?
[21:31:52] … shur down postfix? you never know
[21:31:59] Stef: I think with everything stopped it will be mental.
[21:32:14] … It CANNOT be the apps. vmstat would show it, as would top
[21:32:17] ng: thats pretty much what we saw on the original server
[21:32:37] … networking related?
[21:32:38] Stef: They all use normal mem & CPU -- there's CPU to spare even when the load averages are high.
[21:32:43] … POSSIBLY
[21:32:50] … Just thought that myself.
[21:32:55] ng: though I cant imagine what
[21:32:58] Stef: Possibly DDoSing?
[21:33:19] ng: seems unlikely... but... possible maybe... we changed ip..
[21:33:21] Stef: I dunno, seems doubtful, though, doesn't it
[21:33:50] ng: suppose wh change IP (we have two) ...
[21:33:57] Stef: I'm sure last time when we had access to xentop that the network stats for the VPS weren't out of the ordinary.
[21:34:06] ng: if the problem goes away for a few days then comes back, that would indicate somethjing
[21:34:22] Stef: If I was DDoSing someone it'd be via the DNS name, not a straight IP
[21:34:25] ng: brb .. got food cooking... just need to sort it.. 5 mniutes...
[21:34:34] Stef: K, I'll get a cup of tea.
[21:38:19] ng: back
[21:38:51] … how about installing a fiewwall... block all IPs ecept yours and mine... then run some tests?
[21:39:40] Stef: Yeah, go for it.
[21:40:02] ng: ok... can you get the sheel session open just incase I screw something
[21:40:49] Stef: Heh, that won't help me if you block the world ;) But we've got virtual console access via SSH, so it shouldn't matter :)
[21:41:00] ng: thats what I meant
[21:41:13] Stef: Ah, OK, will just do that now.
[21:41:20] … My IP's 80.21.32.40 btw
[21:41:51] … Oh, I could really do with running away in 20 minutes, sorry! We should be done by then, really, though
[21:42:02] ng: yeah me too
[21:42:18] … lets firewall (if I can get in!) and see if that brings it to sense
[21:42:31] … thgen open up 80 and see what happenbs
[21:43:00] Stef: I'd leave it all firewalled but for our IPs for 5 min. Watch the load avgs
[21:43:10] ng: yeah thats what I man
[21:43:12] Stef: If they're still high, stop all the apps
[21:43:16] … K
[21:43:21] ng: can you get webmin? its just taking ages for me
[21:43:21] Stef: Just sorting out that virtual console
[21:43:36] … Everything's slow :(
[21:45:23] ng: if you have ssh can you shut everything down except webmin
[21:46:19] Stef: K
[21:46:30] ng: its using loads of awap
[21:46:43] … that alone is wrong
[21:46:56] Stef: Yeah, I noticed that too. It's odd
[21:47:12] ng: yet apprently no big processes
[21:47:27] Stef: OK, stopping mysql
[21:48:34] ng: whats your client ip?
[21:48:47] … ok.. nevermind, got it from who
[21:49:21] Stef: I'd enabled syn cookies a while back, btw, which should help against DDoSes. But netstat's not reporting anything particularly excessive.
[21:49:35] … K, mysql and apache are stopped
[21:50:13] … Hmm, load averages are dropping, but I think that's just coincidence
[21:50:18] ng: write here goes with iptables... actually this may not install the config.. but maybe...
[21:50:34] Stef: Hold on with that for a sec
[21:50:47] ng: too late :)
[21:50:57] … well I'll check...
[21:51:38] Stef: It's just that the ssh admin console thing's not working
[21:51:43] … As it's a diff server now
[21:52:38] … Hmm, the server went back to normal, but I really have a feeling that was before mysql and apache were actually stopped.
[21:52:55] ng: dunno.. lets start them again.. no connections now...
[21:53:16] … just your an my IP .. not even localhost is allowed
[21:53:19] Stef: OK, I'll do it
[21:54:33] ng: so... no connections = no problem, it would seem
[21:54:33] Stef: Ah, apache can't start
[21:54:45] … You'll need to add localhost for iptables
[21:54:49] … 127.0.0.1
[21:54:56] … apache2: apr_sockaddr_info_get() failed for ng
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[21:55:04] ng: ok .. with port 80 only though
[21:55:07] Stef: Oh, noticed the hostname changed on reboot
[21:55:23] ng: ghmmm... just thinking, I've borken webmin maybe
[21:55:38] Stef: I don't think we need to be that restrictive
[21:55:48] ng: I nered to connect direct to webmin not via ssh...
[21:56:10] Stef: Ah. Do I need to do it add the IPtables rules from the console?
[21:56:25] ng: ok.. dumb.. I cant cos I rold webmin to only accept localhost logins .. will have to disable iptables first...
[21:56:45] Stef: Hmm, I'm still getting the weird 'run a command and nothing happens for ages' thing
[21:57:01] ng: load avg is almost 0 now
[21:57:45] Stef: (But it was dropping before, unfortunately -think none of this will be conclusive.)
[21:57:49] … root@netgenius:~# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 82.44.120.213 anywhere
ACCEPT all -- 88.107.50.113 anywhere
DROP all -- anywhere anywhere

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[21:58:17] ng: so, add 127.0.0.1 for everything?
[21:59:34] Stef: Yeah, hold on a sec and I'll do that
[21:59:52] ng: I can do it.. already there
[22:00:00] Stef: Ah, cool
[22:00:45] ng: I edit the webmin iptables config file, then use iptables-restore, i.e. iptables-restore /etc/webmin/firewall/iptables.save
[22:00:54] … done now
[22:01:02] Stef: Fair enough
[22:01:35] … Starting apache
[22:01:45] ng: yep, thats given me webmin back again, and pache should start I guess
[22:02:15] Stef: May not if it needs to bind to the actual external IP interface
[22:02:35] ng: true.. ok, I'll do that...
[22:02:53] … um.. actually I think it might start anyway, just not receive anything... try?
[22:03:16] Stef: Ah, OK, it's there
[22:03:24] ng: yeah, thought so
[22:03:39] … so now you and I can access the site
[22:03:47] Stef: That warning may be a problem with the hostname (as it changed on reboot - fairly sure it's not right)
[22:03:58] … Yep, can access the site
[22:04:07] ng: hostname should prob be the www.ksfiom... one
[22:04:33] … though cant imagine it would make a difference... just that apache uses it for messages and stuff
[22:04:34] Stef: Well I could, seems to be timing out now
[22:04:52] ng: working fine for me, and very fast
[22:05:07] Stef: Got a message about cacherouter temporarily then
[22:05:11] … But now it's back
[22:05:29] … (I stopped the memcached service before, perhaps it was tring to use that?)
[22:05:37] ng: hmmm.. dodnt catch the rest of the message?
[22:05:46] … I'll lsee if its in the log...
[22:06:27] Stef: Hm, doesn't matter really
[22:06:45] ng: so, now what?
[22:07:02] Stef: Kill all the iptables rules
[22:07:08] ng: shall we let the people back in, or run ab? I dont think ab will show up anything
[22:07:13] Stef: (Save 'em for a later dat.)
[22:07:17] … date, even.
[22:07:20] ng: kill? I was thiking open up only port 80
[22:07:32] Stef: Sure, that's fine
[22:07:49] ng: so that we are eliminating any kind of DoS attack (unless its at port 80)
[22:07:51] Stef: But I'd prefer a fully open one
[22:07:58] ng: why? i dont understand
[22:08:07] Stef: As it'd show if there was a DoS attack
[22:08:16] … (As in, we were still getting hit)
[22:08:22] ng: not necessarily.. they may have gone away
[22:08:36] … I can log everything other than port 80 etc
[22:08:37] Stef: Yes, I fear that may have been the case already
[22:08:57] ng: id prefer open port 80, let the ppl in, and see if the problem comes back
[22:09:03] Stef: K, go for it
[22:09:16] … Any connections there should show up via netstat tho
[22:09:19] ng: we will need to open up some other things too for mail etc, but later
[22:09:23] Stef: (And they didn't before)
[22:09:26] ng: so, port 80 first?
[22:09:29] Stef: K
[22:10:28] ng: done
[22:10:35] … should start getting connections
[22:11:30] … gonna test connectivity from my other server
[22:11:50] Stef: Me too
[22:11:58] ng: yep, did a wget from there and it worked
[22:12:18] Stef: Yeah, I get through via lynx
[22:12:38] ng: 14 users and 7 guests according to Drupal
[22:12:50] … I suggest we leave it like this until tmorrow?
[22:13:07] … mail will be down, but lets keep it abosoutely basic?
[22:14:01] … what do you think?
[22:14:02] Stef: I'd add the mail back in to be honest
[22:14:12] … if it's no hassle to add 25 outgoing
[22:14:29] ng: I agree, but I'm thiking process of elimination...
[22:14:42] … leave it a day... add mail... leave it another day... add other stuff
[22:15:00] … we can live iwth out mail (its only the notifcations and "contact" messages)
[22:15:01] Stef: I just can't see outgoing-only postfix causing any problems at all
[22:15:11] … But I'm easy
[22:15:26] ng: agreed, but we can't see ANYTHING causing a problem, yet it must be something!
[22:16:07] … and in any case... if the slow-down does oocur now, we know we have a very limited environment
[22:16:40] Stef: True. I'd prefer to add mail back in, and if we still get probs, then remove i
[22:16:42] … it
[22:17:06] ng: um... why not wait just one day? or even 12 hours, till tomorrow?
[22:17:08] Stef: Purely cause I think it's reaaally unlikely that that's what's causing the probs
[22:17:28] … And it'd be good to get the server in a fully functional state
[22:17:55] ng: hang on.. what am I talking about... mail will be working
[22:18:02] Stef: Like I said, I'm easy, just don't want to disrupt the end users
[22:18:05] ng: we're not doing SMTP from outside
[22:18:23] … and all outgoing connections are allowed... I've only blocked incomeing
[22:18:25] Stef: No, we're not. I thought you might be blocking outgoing TCP 25
[22:18:28] … Ahh, OK
[22:18:37] ng: DNS repplies... need those for mail...
[22:18:37] Stef: Well that's alright then :)
[22:18:49] … DNS isn't via this box, is it
[22:18:53] ng: where the DNS server... I'll allow port 53 from there, yes?
[22:19:26] Stef: It's just outgoing DNS on this, isn't it (as in, local lookups)
[22:19:32] ng: try dig.. dont think it will work right now
[22:19:34] Stef: So no need
[22:19:56] … dig on what, the IP? Sure it will
[22:20:09] ng: nope dig google.com
[22:20:24] … we're using xtrahost's DNS I think... look ar resolve.conf
[22:20:35] Stef: Ah, gotcha
[22:21:17] … What are we using for local lookups? pdns, djbdns, dnsmasq, any other local caching servers? Or is it direct
[22:21:35] ng: yeah.. I need to add them, but only for replies from port 53, and only UDP
[22:22:20] Stef: K
[22:24:47] ng: done ... dig is working now
[22:24:56] Stef: Cool
[22:25:08] ng: that should be a fully finctional systen, unless i'm missing something
[22:25:24] Stef: Don't think so
[22:25:44] ng: I'll send a "contact" mail to you...
[22:27:01] Stef: Heh, I just sent myself one for the commandline - great minds, and all
[22:27:08] … Am watching the postfix queue
[22:27:22] … 0EB6F4E091 1477 Mon Nov 3 19:11:31 php(?)ksfiomdepositors [dot] org
(connect to btworld.com[82.98.86.164]:25: Connection timed out)
***********(?)btworld [dot] com
[22:27:57] ng: yeah, cant imagine why that is
[22:27:59] Stef: Nah, outgoing mail doesn't seem to be working
[22:28:03] ng: unless theyve blocked us
[22:28:07] … not at all?
[22:28:30] … ok.. wonder why
[22:28:32] Stef: No, current queue is: root@netgenius:/etc# postqueue -p
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
0EB6F4E091* 1477 Mon Nov 3 19:11:31 php(?)ksfiomdepositors [dot] org
***********(?)btworld [dot] com

7FB664E084* 1052 Thu Nov 6 21:26:22 php(?)ksfiomdepositors [dot] org
***********(?)gmail [dot] com

CD0304E09C* 1668 Wed Nov 5 05:11:51 php(?)ksfiomdepositors [dot] org
***********@talktalk

E91844E071* 1064 Thu Nov 6 21:26:22 php(?)ksfiomdepositors [dot] org
***********(?)netgenius [dot] co [dot] uk

BC7C64E072* 334 Thu Nov 6 21:25:37 ***********(?)mail2 [dot] ksfiomdepositors [dot] org
***********(?)stefpause [dot] com

27FD94E297* 1753 Wed Nov 5 18:15:30 php(?)ksfiomdepositors [dot] org
***********(?)onetel [dot] com

752C64E0B1* 1668 Wed Nov 5 05:11:52 php(?)ksfiomdepositors [dot] org
***********(?)btworld [dot] com

-- 15 Kbytes in 7 Requests.
[22:29:08] ng: restarting postfix... maybe screwed by lack of DNS lookups earlier
[22:29:08] Stef: I'll just try a quick postfix restart
[22:29:15] ng: alrwady done :)
[22:29:17] Stef: Snap :) I'll let you
[22:29:56] … Nah, doesn't seem to be going out
[22:30:03] ng: connections are timing out...
[22:30:05] … um...
[22:30:14] Stef: Hm
[22:30:14] ng: these are outgong connections on port 25 ...
[22:30:20] Stef: Yep
[22:30:34] ng: so wheres the problem... all out going are allowed... or are they? ...
[22:30:52] … yep... they are...
[22:31:08] … what am I missing?
[22:31:15] Stef: hostname related???
[22:31:26] ng: nah.. we getting connection timeout...
[22:31:59] … ih.. Ok.. I'm being dumb!.. I know what it is..
[22:32:07] … we are filtering packets, not connections...
[22:32:21] … so we wont get the replies... like the DNS thing... wait, I'll fix it...
[22:33:30] … there.. should work now
[22:33:43] Stef: Yep, my telnet test just worked
[22:34:16] ng: flussfing mail queue....
[22:34:21] … or flushing even
[22:34:55] … yep... wroking... except those that are giving other errors
[22:35:03] … you should have your test message by now
[22:35:26] Stef: Yep
[22:35:29] ng: I've got mine (sent myself a copy)
[22:35:36] … right sorted.
[22:35:41] Stef: ksfiomdepositors@ng
[22:35:43] … From
[22:36:04] ng: so now we wont even reposnd to ping... maybe should add that too?
[22:36:10] Stef: Do we need to change the hostname from ng to something.ksfiomdepositors.org
[22:36:26] ng: yeah better, though I cant see what would need it
[22:36:32] Stef: Ping/ICMP's a possible attack vector
[22:36:43] ng: eactly... I can limit it by packet rate...
[22:36:43] Stef: That's the classic original DDoS
[22:37:01] … I'd say leave it off for the moment. Don't need it
[22:37:17] ng: ok, only problem for network monitors etc
[22:37:25] … but ignore that for now
[22:37:41] Stef: True. My monitors just hit port 80 with a HEAD lookup
[22:37:54] ng: that will work then .. I have one doing that too on my server
[22:38:12] Stef: Cool. Yeah, let's call it a night, then?
[22:38:18] ng: ok, DoS seems unlikely, but this should make it near impossible, in case it is happening
[22:38:45] … well.. they can DoS us though lots of web page loads... but our ab tests indicated that the system holds up well, true?
[22:38:54] Stef: Aye. I really don't think it is DDoS, purely because kept doing netstats
[22:39:03] … And they'd show up there (I think)
[22:39:31] ng: I agree... but it could be some kind of scanner, or other bot
[22:39:38] Stef: Even so, it'd still show up
[22:39:42] ng: maybe not for DoS purposes, only with that effect
[22:40:18] Stef: We'd still be able to see it though. Dunno, is all just really frigging ODD :(
[22:40:24] ng: shall i log pacjets? easy, just one more line... disk space might be the only issue (we've got gigs though.. I can rate-limit the logging)
[22:40:34] Stef: I'm easy
[22:40:52] ng: maybe tomorrw.
[22:41:01] … see what happens with this set up.
[22:41:12] … so, are we done?
[22:41:30] Stef: Yeah, I reckon so. All very, very strange stuff! :)
22:41:50] … Right, night fella.
[22:41:52] ng: "207 guests online." !!!!
[22:41:57] Stef: :)
[22:42:08] ng: havent seen iot that high since the liquidation meeting
[22:42:33] … and its still mega fast
[22:42:40] … just like it should be
[22:42:51] Stef: /bin/netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | more
[22:43:01] … That should give an ip breakdown of connections
[22:43:39] … I'm top with 7 at the moment
[22:43:54] ng: not so many.. but not surprising.. "guest" are just unique ips who have loaded a page within the last 10 minutes
[22:44:13] Stef: Aye. Anyway, I better run and eat some tea, I forgot to cook again!
[22:44:19] ng: lots of reading, so the connection would time out-- what do we have for max-keepalive? one minute?
[22:44:36] Stef: Something like that, yeah
[22:45:10] ng: ok, figures then.. gret good work! have tea!
[22:45:23] … 9:45, not exactly team time!
[22:45:30] … or tea time
[22:45:53] Stef: Heh, I know! :) Cheers , take care fella - night
[22:46:27] ng: gonna post all this into the IT group so ppl can see how hard we work for them! ... talk to you tomorrow, take care.


0
Your rating: None

Comment viewing options
Select your preferred way to display the comments and click "Save settings" to activate your changes.

It stinks of no natural justice

  • sami
  • 10/10/08 31/05/09
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 10:48

Yes I understand, it makes me angry too. Many years of working hard and paying 40% tax. Most years I paid several tens of thousands of pounds. I think a lot of us on here are the same - successful and hard working.

It all sounds like a lack of natural justice. I refuse to talk about 'not winning this fight' - I do not entertain the idea so will not mention it.


ng & stef

  • sad
  • 10/10/08 31/10/09
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 00:56

good god, i thought international banking was difficult to understand til i read this......are you speaking/typing english!!! seriously, thanks for everything. I am of course still waiting to be told where to send my money for fantastic service and support you are offering....Thanks again,


Tech team

  • shafted
  • 10/10/08 12/12/09
  • a depositor
  • Offline
  • Thu, 06/11/2008 - 23:22

Thanks NG and Stef, you have and are doing an amaising job with minimal down time, i am not sure if i have missed a posting on how to contribute to your work for us all?


IT - thanks

  • Knife Edge
  • 10/10/08 31/05/09
  • a depositor
  • Offline
  • Thu, 06/11/2008 - 23:14

ng and Stef - you guys really have set up a fantastically valuable tool and resource here, and I can't commend your work highly enough, it really is hugely appreciated by everyone.

I for one rely on this site as a primary source of information and support on at this worrying time, and I'm fully aware of the fact that I couldn't do what you've done, and how time-consuming what you're doing in the background must be. So, a big thank you!


DOS attack

  • P.Taylor
  • 19/10/08 31/05/09
  • not a depositor
  • Offline
  • Thu, 06/11/2008 - 22:53

I must admit that was my first thought (Denial of Service attack) when the website first slowed up a week or so ago. If it's not, it's not, but there are plenty of people out there who like to make mischief, and maybe some with a logical motive.

Nice work anyway (ng and Stef). Good luck.


If it is DoS related, that might be very sinister indeed

  • ng
  • 11/10/08 31/12/20
  • a depositor
  • Offline
  • Thu, 06/11/2008 - 23:52

Hi PT.

Do you really think so? But random or targeted? There is something interesting to consider... every time we have moved servers (change of IP address) the site has been fine immediately afterwards. In this case, we first set up with the current hosting company a couple of weeks ago. The site worked perfectly for a few days then started to go horribly wrong. We eventually moved away temporarily (change of IP address) and a couple of days later back to xtrahost but on a different machine (different IP address again). That was a week ago. Up until yesterday it's been working perfectly as far as I know. Yesterday I noticed a few of the "very slow" periods, and today the site became unusable again... the server running extremely slowly, apparently very busy doing nothing. All this could be considered to be indicative of a possible DDoS attack - change of IP each time making them lose us for a while until they updated their target IP.

For those that would like to know - from Wikipedia ...


A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a computer resource unavailable to its intended users. Although the means to carry out, motives for, and targets of a DoS attack may vary, it generally consists of the concerted, malevolent efforts of a person or persons to prevent an Internet site or service from functioning efficiently or at all, temporarily or indefinitely. Perpetrators of DoS attacks typically target sites or services hosted on high-profile web servers such as banks, credit card payment gateways, and even DNS root servers. (more...)

We keep saying it's unlikely because the questions would be who and why? So, let's not be over-dramatic, it is unlikely. But if it were a DoS, and targeted ... well - that really would be something new for our legal team, let alone the media. But, really, not at all likely.


DOS attack

  • P.Taylor
  • 19/10/08 31/05/09
  • not a depositor
  • Offline
  • Fri, 07/11/2008 - 10:05

ng, I'm not an expert on this topic (DOS attacks) but yes, targeted. Perhaps it is unlikely but if you think of the purpose of this website... ? A lot of money at stake and a very public story that involves countries and governments and vested interests in those countries and governments who wish it would all go away, except, of course, the depositors who have everything to gain from the continuation of this website. I only suggested this because the reason for the slowup appeared to be a mystery, but I would imagine a DOS attack would be detectable by the server administrator, would it not?


Does Big Brother Think We're A Threat to UK Bank Stability?

  • VikingRaider
  • 10/10/08 31/05/09
  • unspecified
  • Offline
  • Fri, 07/11/2008 - 00:44

Ng you and your associates are doing a grand job, but you must have realised by now that we are probably THE biggest fly in the ointment for HMG's bank restabilisation strategy simply because we exist and, consequently, undermine confidence in British banks.

The Gilbert-andSullivan comic opera into which HM Treasury has transmogrified is threatening to turn HMG into an international laughing stock. This spooks institutional investors who appear to be on the brink of scuttling the HBOS-Lloyds merger -- a gaff that would make HMG appear like the St Trinians' gals hockey B team.

You may rest assured that the masters of counter-spin are working around the clock to fuel doubts in the British banking system (hence $US1=£1.57 and today's 1.5percent BoE rate cut) and the godawful racket that contributors to this website, Douglas and Reykjavik are stirring up at Westminster, in Whitehall and in Threadneedle Street means that we should entertain few doubts that WE ARE BEING WATCHED -- by GCHQ Cheltenham in particular. This is an entirely unforeseen tactical challenge for the Gaslighters of Cheltenham, however; their hands are tied by the fact that none of us (I hope) are 'proscribed individuals' and that so many of us are not resident in the United Kingdom. The more spanners they try to throw at you, the more it confirms that our message is hitting the bullseye.


Great work

  • Peasant
  • 10/10/08 31/05/09
  • a depositor
  • Offline
  • Thu, 06/11/2008 - 23:34

I noticed earlier today that, IIRC, there were 62 members online and 186 guests. Also noticed kept dropping and having to re-login. I thought then that summat was up!

As it happened I hardly noticed the problems this evening as I was on another forum that had similar problems a few months ago which were resolved by changing the software. The original, which they had been using for 3+ years slowed to snail pace when there were many online at once.

Well done to both of you for sticking with it and winning through.


Thanks for the info, but

  • ng
  • 11/10/08 31/12/20
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 00:00

Thanks for the info, but there's absolutely no indication that the problem was related to number of on-line users. When we have lots of people on-line (maybe 300 or more) you may see things going a bit slowly, but only in terms of it taking maybe 5 or 10 seconds to load a page rather than 2 or 3.

After our work tonight the site had around 300 users on and was fast. Before we started it had been virtually dead. The problem may be somehow related to number of uses, but it's not a simple relationship.


Superb work guys

  • Done like a Kipper
  • 10/10/08 n/a (free)
  • a depositor
  • Offline
  • Thu, 06/11/2008 - 23:22

Just to say that I really appreciate all your efforts to keep this 'lifeline' up and running. I'm sure that I also speak for all those who find themselves in this quite dreadful situation. Without your efforts plus those of the London and IOM teams we wouldn't be where we are today!

Great work!


Great Work Team!

  • caledonia
  • 14/10/08 30/09/09
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 09:42

Yes,GREAT WORK TEAM and please, please keep it up - so many of us depend on you - you are our lifeline. But don't underestimate Cheltenham ........ one of those situations you never think it is going to happen to you - but maybe it just has. Just goes to prove that all your/our efforts are working and we are now a force to be reckoned with - we are not a voice in the wilderness and are not going to be swept away under the carpet. So keep it up everybody, because they ain't seen nothing yet! Well done!
Yours, a very grateful grandmother whose typing abilities having greatly improved during the last three weeks although still have to have several 'goes' at the Captcha!


well said Caledonia

  • bellyup
  • 10/10/08 09/01/10
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 10:15

Thank you ng and stef
I don know what we would do without you.
I practically live on this site hope I'm not making it slow down!


I noticed all this slowing up

  • NOT darling
  • 06/11/08 31/05/09
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 10:45

I noticed all this slowing up and difficulty getting onto the Action Website, yet I could log into anything else really quickly. I kept getting refused when everything else I logged onto was fine. Like you said the amount of people should cause very little problems. Alot of people are using this Action Group as a lifeline and it is growing everyday so great work and thank you so much.


Bad Mood today

  • caledonia
  • 14/10/08 30/09/09
  • a depositor
  • Offline
  • Fri, 07/11/2008 - 10:36

I'm in a bad mood today .... which is making me very ANGRY and I just have to voice my feelings ... sorry!
We are in this big time - a life times hard earned UK tax paid (at 60%) savings. Took early retirement due to ill health as an expat (therefore no pension coming in) and STILL paying HMG tax on our interest (or did). If we go down because of this I will make sure we go back and throw ourselves on the state - after all, we have paid enough into it over the years and until now, have taken nothing out. Time to change ! They think they want their cake and eat it ? WRONG - just watch out Gordon and his "Darling" !!
I feel better now ...............