Forums in Slow Motion/Your reports needed

[hostingtechs]

Hopefully there will be no downtime at all in the future. This is what I meant by indefinetely.

Posted by céline (here)
i just tried to send a pvt msg and got an error msg...becuase i dont save msg's i have no way of knowing if it sent.

Can you elaborate on what the error message said? Thanks. :-)

[Celine]

hmm its not there now..

It was one of the "error" messages that we have been getting in the last 2 days.

[Celine]

it is happening again...at intervals...

Error 503 Service Unavailable

Service Unavailable
Guru Meditation:

XID: 1212575536
Varnish cache server

[hostingtechs]

Posted by céline (here)
it is happening again...at intervals...

Error 503 Service Unavailable

Service Unavailable
Guru Meditation:

XID: 1212575536
Varnish cache server

I went ahead and added more buckets to the Varnish cache as well as rewrote the hashing algorithm. Hopefully this will fix it. Let me know if you see more errors.

[Anchor]

Why would the quality of the cache affect the availability of the underlying service ?

What are the criteria for availability - and - is it availiability to the end user or the caching layer ?

[hostingtechs]

Posted by John (here)
Why would the quality of the cache affect the availability of the underlying service ?

What are the criteria for availability - and - is it availiability to the end user or the caching layer ?

As far as the first one, the caching layer is responsible for all connections between users and the site, so if the cache runs out of buckets (connections and their associated cached content), it leads to 503/Guru Meditation errors. This reduces load on the webserver by keeping the most requested items handy in memory.

Can you clarify what you mean by the second question? I'm afraid I don't understand.

Alex

[Celine]

Seems overly complicated for a simple operation...

[hostingtechs]

Unfortunately, it does. However, running software such as vBulletin is extremely load/resource intensive, and as such, requires caching, etc to make the site function smoothly.

[hostingtechs]

Load Test Results for those interested. This just finished. Feel free to ask if you have questions.

[Anchor]

I am finding the forum slow still.

New posts function seems quicker than I remember it.

"Post Quick Reply" is slow but is how I posted this

"Reply" is unusuable the times I tried it in the last 4hrs

"Reply with quote" is unusable any time I have tried it, but not today

John..

[Anchor]

Posted by John (here)
I am finding the forum slow still.

New posts function seems quicker than I remember it.

"Post Quick Reply" is slow but is how I posted this

"Reply" is unusuable the times I tried it in the last 4hrs

"Reply with quote" is unusable any time I have tried it, but not today

John..

Reply with quote appears to be working

John..


Update: edit post is working, but it took a lot of time to load the screen for editing (4-5seconds)

[hostingtechs]

Posted by John (here)
Reply with quote appears to be working

John..

Glad to hear we're making progress . How about Post Quick Reply and Reply with Quote in terms of speed?

--
Karl

[Etherios]

I dont know if its me ... but 3 days in a row now i find the forums down for hours. anyone else have same problems?

[hostingtechs]

Posted by Etherios (here)
I dont know if its me ... but 3 days in a row now i find the forums down for hours. anyone else have same problems?

Yes, they did have a lot of problems recently but everything should be fixed as of yesterday.

--
Karl

[Anchor]

I was not in a position to test this last night - today I seem to be able to post, but the forum is still sub-optimal in terms of speed. I cannot show my tracert from where I am but I am going via from Australia'n office - via fast Corporate network to a proxy elsewhere in Asia (sorry cant say where) - Internet -> Avalon.

Sounds complex but PA1 was pretty fast at its best with this setup.

Which of the "layers" is now the slowest?

John..

[hostingtechs]

Posted by John (here)
Which of the "layers" is now the slowest?

I am a little confused as to what you are asking?

--
Karl

[Anchor]

Posted by hostingtechs (here)
I am a little confused as to what you are asking?

--
Karl

I have assumed that your architecture is organized in layers. With the storage hardware at the very base, a database engine, a forum engine, an httpd layer, a load balancer etc etc. Which of these is the slowest ? Has the interface between all these layers been appropriately instrumented and monitored for health, efficiency and effectiveness ?

When I asked earlier in the thread related to the "530 service not available" message I was really asking Which service - ie which "layer" was letting you down.

Perhaps I am using different terminology - would "tier" be more appropriate?

John..

[hostingtechs]

Posted by John (here)
I have assumed that your architecture is organized in layers. With the storage hardware at the very base, a database engine, a forum engine, an httpd layer, a load balancer etc etc.

That is correct.

Posted by John (here)
Which of these is the slowest ? Has the interface between all these layers been appropriately instrumented and monitored for health, efficiency and effectiveness ?

Neither of them are the slowest. They're all running at equally the same speeds. And yes they have been monitored. The load storm test that Alex posted was a great performance test, and the system stayed fast and did not go down during this test.

Posted by John (here)
When I asked earlier in the thread related to the "530 service not available" message I was really asking Which service - ie which "layer" was letting you down.

That would be varnish; however, we think we have fixed that issue.



--
Karl

[Anchor]

Today I find the forum is running well - just thought you should know

C:\Windows\system32>tracert projectavalon.net

Tracing route to projectavalon.net [206.217.197.208]
over a maximum of 30 hops:

1 89 ms 98 ms 99 ms BigPond.BigPond [10.0.0.138]
2 25 ms 24 ms 25 ms 172.18.112.75
3 25 ms 25 ms 25 ms 172.18.65.26
4 26 ms 25 ms 25 ms 172.18.239.9
5 47 ms 25 ms 25 ms Bundle-Ether10.ken39.Sydney.telstra.net [203.45.17.17]
6 24 ms 24 ms 27 ms Bundle-Ether6.ken-core4.Sydney.telstra.net [203.50.6.145]
7 24 ms 26 ms 25 ms Bundle-Ether1.pad-gw2.Sydney.telstra.net [203.50.6.29]
8 27 ms 25 ms 24 ms TenGigabitEthernet2-0.sydp-core02.Sydney.reach.com [203.50.13.50]
9 183 ms 183 ms 184 ms i-2-1-1.wil-core02.bx.reach.com [202.84.140.38]
10 178 ms 179 ms 178 ms i-3-0-0.paix-core02.bi.reach.com [202.84.140.90]
11 179 ms 178 ms 178 ms i-1-1.paix01.bi.reach.com [202.84.251.70]
12 * * * Request timed out.
13 214 ms 214 ms 214 ms vl53.ar1.sfo1.us.nlayer.net [69.22.143.134]
14 230 ms 230 ms 230 ms te1-1.ar1.slc1.us.nlayer.net [69.22.142.96]
15 267 ms 266 ms 264 ms xe-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.101]
16 263 ms 266 ms 268 ms ae1-40g.ar1.ord1.us.nlayer.net [69.31.111.134]
17 265 ms 266 ms 267 ms ip-69-31-110-46.nlayer.net [69.31.110.46]
18 no-ptr.midphase.com [206.217.197.208] reports: Destination protocol unreachable.

This isnt much different from how it was on a bad day.

John..

[hostingtechs]

Posted by John (here)
Today I find the forum is running well - just thought you should know

C:\Windows\system32>tracert projectavalon.net

Tracing route to projectavalon.net [206.217.197.208]
over a maximum of 30 hops:

1 89 ms 98 ms 99 ms BigPond.BigPond [10.0.0.138]
2 25 ms 24 ms 25 ms 172.18.112.75
3 25 ms 25 ms 25 ms 172.18.65.26
4 26 ms 25 ms 25 ms 172.18.239.9
5 47 ms 25 ms 25 ms Bundle-Ether10.ken39.Sydney.telstra.net [203.45.17.17]
6 24 ms 24 ms 27 ms Bundle-Ether6.ken-core4.Sydney.telstra.net [203.50.6.145]
7 24 ms 26 ms 25 ms Bundle-Ether1.pad-gw2.Sydney.telstra.net [203.50.6.29]
8 27 ms 25 ms 24 ms TenGigabitEthernet2-0.sydp-core02.Sydney.reach.com [203.50.13.50]
9 183 ms 183 ms 184 ms i-2-1-1.wil-core02.bx.reach.com [202.84.140.38]
10 178 ms 179 ms 178 ms i-3-0-0.paix-core02.bi.reach.com [202.84.140.90]
11 179 ms 178 ms 178 ms i-1-1.paix01.bi.reach.com [202.84.251.70]
12 * * * Request timed out.
13 214 ms 214 ms 214 ms vl53.ar1.sfo1.us.nlayer.net [69.22.143.134]
14 230 ms 230 ms 230 ms te1-1.ar1.slc1.us.nlayer.net [69.22.142.96]
15 267 ms 266 ms 264 ms xe-2-1-0.cr1.ord1.us.nlayer.net [69.22.142.101]
16 263 ms 266 ms 268 ms ae1-40g.ar1.ord1.us.nlayer.net [69.31.111.134]
17 265 ms 266 ms 267 ms ip-69-31-110-46.nlayer.net [69.31.110.46]
18 no-ptr.midphase.com [206.217.197.208] reports: Destination protocol unreachable.

This isnt much different from how it was on a bad day.

John..

Glad to hear it is running well.

I do notice there is no Cogent in your route though and I think there was before, and they tend to cause A LOT of issues.