Server status, October 24

User avatar
ChrisW
Joined: January 31st, 2013, 10:58 pm

October 17th, 2016, 1:20 am

.Update October 24


Server status
Most servers are stable now, but we still have some lingering issues on C2, C8, C10, C11 and C12

C11 has some boards with bad indexes. This means that when you have a topic with multiple pages, you might see en empty page on the last number of the pager. If this is happening to your board, please post in this topic. This might also an issue on C8 and C10

C2 and C12  are lagging occasionally. There are some domains with 'bad' topics (Mostly topics made immediately after the outage.). When said topics are viewed or posted on, it can ccause the server to start lagging. Each time this happens, the domain is cleaned up, and those lags will become less frequent as the days go by. It is very important that you report those lags in this topic here.

Board moves:
Since we have the brand new C17 which is in excellent shape, we might as well move many boards to it. Please continue to post your links at http://yukusupport.yuku.com/topic/5295/New-server to get in line for a move.

Please check http://yukusupport.yuku.com/topic/5377/Board-moves# to find out about the status of the moves, I will be updating that topic throughout the day.

Help Desk:
I am a bit backlogged there again, please allow a little extra time for ticket replies. All tickets will be replied to.

Thanks again for the incredible patience you have all shown throughout.


. Update October 20 at 4:20 PMHello everyone,Here is today's update:- C6: We are still seeing issues with avatars and usernames not working, the fix for this is still in progress. So sorry this is taking a while.- C8 is having an issue with the boards going back in time occasionally. This is being investigated.- Many C10 boards have been moved to C17, and I'll be putting more in the queue today.- We also have many non C10 boards listed who want to move to C17. Those too will be moved, after all servers are stable.- Don't forget to redo your forum and topic subscriptions if you had any.- A slight lag was reported on C5, it is being looked at now.- Most gold boards have been credited for the outage. If you have a gold board and you have not been credited yet, please submit a ticket by hitting the red button here at support.- Help desk is now fully caught up and ticket replies should happen with the same speed as usual.- If you still see something out of place that has not been mentioned here, or if you have asked a question somewhere that has not been replied to yet, please post in this topic here.. Update October 19 at 12:40 PM Eastern Time

Hello everyone,

Let's see where we are today.

1) A fix is running on C6 for the missing usernames and avatars. This will still take a few hours.
2) C10, which was hit the hardest, is not stable yet. Please keep posting your board links at http://yukusupport.yuku.com/topic/5295/New-server
Any C10 boards listed there get absolute priority.
3) I am currently preparing the domain ids to be fed to the move script. As soon as I'm done with that, the boards will be moved. I will post in the new server topic when that happens, who is moved, etc..
4) C8 is on the list to be looked at.
5) I have testing scheduled on C11 and C12
6) Important notice. I will likely send out a global announcement about this, but for the time being, I'll post it here. The outage caused major issues with topic / post subscriptions. Our technical staff had to dump all subscriptions. If you have subscriptions to boards, forums, and topics, you will need to redo them. Even if you received notifications today still, your subscriptions will be gone now. I am really sorry about the inconvenience. Here too, it was a choice between the lesser of two evils.

As usual, stay tuned and I will keep you updated.

======================================================


Update October 18 at 8:35 PM Eastern Time


C10 is not fully stable yet at this point. I have a good number of boards queued to be moved off of there.Some boards will see an accounts replication lag. This means that some usernames and avatars will not be showing. If you see this, please do report it in this topic.

Also, please continue to report any lagging issues and feel free to add your boards to http://yukusupport.yuku.com/topic/5295/New-server#
23 active boards have been moved from C10 to C17 and are doing well. The moves, however, are taxing on the server, so remaining boards on C10 may experience occasional downtime.


Help desk:  The tickets were caught up but that was short-lived. Any urgent issues, please make sure to report them here in this thread, rather than via a ticket.I will continue to update this topic tonight, please check back for further news.Thanks :)

. Update October 18 at 3:38 PM Eastern Time

We need to give high priority to moving boards off of C10. While this is being done,  C10 is being kept off line.
Please list your boards at http://yukusupport.yuku.com/topic/5295/New-server


. Update October 18 at 1:50 PM Eastern Time

All servers should be operational. However, C10 and C12 are not out of the woods yet, and the more boards we can move, the better, so please continue to list your boards in the http://yukusupport.yuku.com/topic/5295/New-server topic. Anyone listed in that topic will be moved with the first batch and will be out of the woods sooner.

Please report ANY and every lag (disappearing posts) immediately, it's VERY important that we know of any lags. Don't forget to add a link to your board when you post a lagging issue.


.Update October 17 at 1:14 PM Eastern Time

The maintenance flags are being lifted from the C10 boards. Please keep in mind that indexes are being rebuilt, so you might experience slowness at times. Also, new members might now show on your member list for a long time (until replication catches up). They will also have their name missing from posts.

Don't forget to list your board at http://yukusupport.yuku.com/topic/5295/New-server if you'd like to have it moved to 17, which is as stable as .. well as something real stable :)

=========================


.Update October 17 at 1:10 PM Eastern Time

C10 is off line for further repair.  You should see the maintenance flag on your C10 boards now. (Make sure to refresh if you don't see it).
If your board is down and you don't see the maintenance flag, please report it in this thread here ASAP.

More on C10 boards: As soon as the maintenance is done, we are going to move a good number of boards from C10 to C17. If you know your board is on C10, or even if you're not sure, please post your link in this topic: http://yukusupport.yuku.com/topic/5295/New-server

I will grab the links from that topic and queue them for the move. C17 is the same server yukusupport is on.

===================================


.Update October 17 at 11:55 AM Eastern TimeThe C10 server is currently undergoing an unscheduled maintenance. Tentative ETA for the server to be brought back on line is noon Pacific time.
.Update October 17 at 11:28 AM Eastern Time

It appears that C10 is down. I have alerted the technical staff, and I am waiting for an update at this time. As always, I will post the update here as soon as I have it.
Thanks


.Update October 17 at 1:10 AM Eastern Time

Several servers will need to be restarted to finish a maintenance. In this order: C2, C10, and C12. Boards on the server will be down briefly during the reboot.


. Update October 17 at 2:12 PM Eastern Time

Hello everyone,
While we are not completely out of the woods yet and some repairs are still being done in the background, the boards should be fine for the most part. Some glitches are still expected, but it's very important that you report any glitches here in this topic. We want to check out every report.

Also, if you don't see any more issues, feel free to report that too, so that we know.

Help desk:  The help desk is still backlogged but I'm catching up. I expect the backlog to be cleared in 4 hours at the most. For anything urgent, please post in this topic here, rather than submitting a ticket.

Not really related to the outage, but anyone who would like to see their board moved to our newest server, feel free to leave your link in this topic:
http://yukusupport.yuku.com/topic/5295

Thanks again for all your patience. I cannot tell you often enough how much this is appreciated. You've all been super.


.Update October 17 at 11:40 AM Eastern Time

Some boards might show a white page briefly. Don't panic, the blizzard will be gone shortly.


Update October 17 at 11:10 AM Eastern Time

C12 is up. However, there may be a lag at this time.

At this point it is extremely important that you report lags in this thread here. Please report the lag, the exact time you last noticed the lag, and a link to your board.

If you see a maintenance page on your board also report it please.

Also, I am looking for a volunteer who will be around for the next few hours to monitor something and has access to skype. If you're the one, please PM me.


.Update October 17 at 10:30 Eastern Time

Some good news. C10 boards are back. Please note that a lag is expected at this time during the recovery process.
If you have a board on C10 and you see the maintenance page, please let me know by replying to this topic here.

C12 is close to being done.

C11  was restarted which should help with the lag.

C2, 4, and 8: Please report if you see a lag now.

=============================


.Update October 17 at 9:45 Eastern Time

Good morning, everyone. My apologies for not updating sooner, things have been extremely hectic. I am hoping to give regular updates now throughout the day and into the night if need be.

First, please note that ticket replies will take a bit longer than usual, due to backlogs at the help desk. Everyone will receive a reply, however.

C10 was the most affected by the outage and the fix still has many hours to go. To speed this up, our technical staff is going to restore all boards on C10 from backups, this should shorten the repair time.  I cannot give an exact ETA, so sorry about that.

C12: Once C12 comes back on line, a lag will be expected for a little while.

C6: We had several boards down on C6 last night, all should be good now.

C2, 4, 8, and 11: Replication on those servers is being monitored. If you see disappearing posts on those at this time, please do let us know immediately. It is a possibility.

All other servers, including profile servers, should be up and running and fine. If you see anything out of place, please don't hesitate to let us know.

Once again, our sincere apologies for the current issues. We hope to get you all up and running soon.


==========================================



.Update at 10:45 PM Eastern Time

C10 and C12 will be brought back on line in this order, first C10 and then C12. When exactly that will happen depends on a few things, so at this time I don't have an exact ETA, sorry. I will post here as soon as I know more.
A very small amount of posts made on C10 while the server was up briefly, will be lost, but no posts from before the outage will be missing.

C2: Here too, a very small amount of posts made after the server came back on line might go missing, but is should really be a small amount.

C4 is currently resynching, which will make the lag disappear.

=============================

Hi everyone,
I am going to post this in the regular forum instead of in the announcement forum so that you can reply if you wish to do so.

As you know, we had a long server outage that was caused by hardware issues in the Network Operations Center.
After the original issue was fixed, maintenance had to be done on all servers to make sure that nothing was broken.

Some of the maintenance is still ongoing at this hour.

What to expect:

C10 and C12 are still down for maintenance. I will edit this post with updates as soon as I have them, so please check back often.

Search has not been restarted yet, so you will likely find search down.

Boards on C11, C8, and C4 have reported lagging. C4 and C8 are being looked at. C11 has a confirmed lag which will become less and less, and disappear completely in just a bit.
If you see posts disappearing and you are not on C11, please do report it in this topic. You don't need to re-post the ones that disappear, they will return on their own once the server catches up.
Also, if you see anything out of place, please post in this topic as well. Don't forget to leave a link to your board, this will speed up help.

Last but definitely not least, I know that this has been a rough weekend on everyone. The patience you have shown throughout this has been absolutely amazing. Thank you very much!

On behalf of our team, I want to sincerely apologise for the inconvenience this has caused to the communities. Please know that we are determined to get all communities back on line and working properly as soon as humanly possible.

Don't forget to check back here for updates. The time in the title will reflect the time the topic was updated.

Thank you. 
Last edited by ChrisW on October 24th, 2016, 8:20 pm, edited 18 times in total.
"One can see clearly only with the heart. Anything essential is invisible to the eyes"~~Antoine de Saint-Exupéry~~
Reply

User avatar
GlitterRock
Joined: March 13th, 2013, 1:19 pm

October 17th, 2016, 1:43 am

Thanks Chris, and thanks to everyone working on the fix. :)
"We all change, when you think
about it. We're all different people all through our lives. And that's
okay, that's good, you've got to keep moving, so long as you remember
all the people that you used to be..."
Reply

User avatar
StormyRed
Joined: February 16th, 2013, 3:47 am

October 17th, 2016, 1:43 am

We're still down, so I guess we're on one of those servers in maintenance.....all of my other groups are up and working normal.....how do I check what server SBOS is on ? 
                                                                                            
Reply

User avatar
DoodleMuffin
Joined: February 16th, 2013, 12:57 am

October 17th, 2016, 1:52 am

Thank you so much for all your hard work on keeping us all updated Chris it is so much appreciated.
Hugs Lynne.
Reply

User avatar
Mrs Peavey
Joined: October 19th, 2013, 9:15 pm

October 17th, 2016, 1:56 am

My forum, Peavey's Place, has new threads and replies posted today that have disappeared.  I'm not sure, but I think Peavey's Place is on server C4.
The only thing necessary for the triumph of evil is that good men do nothing


Reply

User avatar
GarlandTex
9
Joined: April 15th, 2015, 5:31 pm

October 17th, 2016, 1:59 am

horseracingtoday.yuku.com is still down. I can access the "Welcome" page but after that all I get is a blank screen. Also, if I try to access calltothepost.yuku.com I have the exact same result - white screen, blank, no clocking Not sure what server either are on.Ernie aka GarlandTex
"My bets never lose, they just don't win."


Reply

User avatar
bixblitz
Joined: April 24th, 2013, 11:21 pm

October 17th, 2016, 1:59 am

Thanks for the update!

We're on C2 and all new posts from today are disappearing.
http://kellyclarksonexpress.yuku.com/
Reply

User avatar
LewisNixon
Joined: October 29th, 2013, 9:02 pm

October 17th, 2016, 2:05 am

but still have lagging just as bad as it was 3 hours ago. No improvement to speak of. You have to make a new post to see what's been posted.
Reply

User avatar
GarlandTex
9
Joined: April 15th, 2015, 5:31 pm

October 17th, 2016, 2:20 am

LewisNixon wrote:but still have lagging just as bad as it was 3 hours ago. No improvement to speak of. You have to make a new post to see what's been posted.
At least you can make a post...
"My bets never lose, they just don't win."


Reply

User avatar
Rosemary1947
Joined: March 8th, 2013, 4:14 am

October 17th, 2016, 2:46 am

I have a board on c2 also...1 post from today disappeared. All my other boards are back up and working fine except of course for my board on c10. Thanks Chris and everyone who worked so hard to repair Yuku.Imagehttp://birdssquirrelsandgardens.yuku.co ... eral-Board
Reply