Update November 21 at 5:45 PM EST

User avatar
ChrisW
Joined: January 31st, 2013, 10:58 pm

November 1st, 2016, 4:08 pm

.Update November 21 at 5:45 PM EST

C20 experienced an issue with its memory. Some boards are being moved to a different server (also a new one), to make sure this does not happen again.
I am very sorry for the continued troubles. Tentative ETA to have it all resolved will be 9 days (when all boards are moved to the new server location). This does NOT mean that things will be down for 9 days, but  slowness and occasional outages can be expected during the 9 day period.
Once we're all in the new location those problems should be no more.


. Update November 16 at 6:50 PM EST

All boards are up and running and server is steady.

===================================

. Update November 16 at 5:34 PM EST

C20 is currently undergoing an unscheduled maintenance. Our technical team is working to get the server back up. Please know that everything is being done to ensure that the boards can come back on line as soon as possible. I know that this has been a long outage, and I really appreciate your patience and understanding. We are going through a rough stretch with the data moves. Outages are still expected until all boards are on the new servers, a process that is getting the highest priority at the moment. Please also see the updates of November 9 and November 15 for more details.

Thank you!

============================


, Update November 16 12:41 AM EST

We are still experiencing DDoS attacks. Our team is working to migitate the attacks. Most boards should be accessible at the moment.


. Update November 15

Hello everyone, it's time for another server update. 

Today's downtime
You might have noticed downtime today, unfortunately this was caused by a DDoS attack. All boards should be back on line at this time.

Up next:
The new servers are set up. As you know from my previous update, boards on the new servers will run slow until all boards are moved. This is because some data is being pulled from the old servers still and it is slowing those boards down.

Next: A few boards will be moved to the new servers (other than C20) and the servers will be tested thoroughly. Immediately afterwards, board moves will begin.

What to expect:
There will be downtime during the moves. This is to make sure your board's data is preserved properly during the transfer and nothing will get damaged.
Boards on the new server will be slow, but should not experience any more downtime, unless the occasional scheduled maintenance which will be announced in advance. Scheduled maintenance is done during off-peak periods (e.g. in the middle of the night EST and PST)
Search will not work immediately on the moved boards.

After the move.
Once all boards are moved, and the old servers are turned off, you will notice an incredible improvement in loading speed. This is the ultimate moment we're all waiting for.
ETA: I am extremely reluctant to even mention this, because it is a long road between here and there, and bumps as well as delays should be expected, but I know that you're all eager to know when this will be behind us.
Contrary to my habits, I am going to give a tentative ETA: two weeks. Again, bumps and unexpected delays can prolong this journey but rest assured that our entire staff is handling this project with the most urgency.

=============================


,UPDATE November 9 at 5:48 PM EST

As you know, several servers went down last week. Instead of trying to patch up the old servers, our team decided to put their already planned upgrade into a different gear and set up a brand new server. This server is now up and running with fully upgraded hardware.

The good: The server is several times bigger, and much more stable than the old servers could ever have been. Your boards will be safe there for years to come.

The bad: This server is in a different location, the future home of all Yuku servers. In itself, that's not a bad thing at all, but the problem you are experiencing is that the old servers are still in the old location. Data gets pulled across, causing a slower browsing experience than what you're used to.

Next up: More new servers are being added that will run alongside the currently lone server in the new home. All boards will be moved to the new servers (except the ones that are already there of course).
Once everything is moved over, the slowness will be a thing of the past because there will be no more dragging data from old servers far away.

Will it involve downtime? Yes, it will. This is not to annoy you, but to make sure that all data is safe during the move. The difference with the downtime from last week is that you will get plenty of advance warning, and the moves will go much, much faster.

Right now you're all still a bit in the wilderness, but you are on the way to a brand new server setup of which I can gladly guarantee you that they are our network's promised land. I am asking you to hang in there for a bit, until all the server works are behind us and I truly am sorry that this is causing inconveniences across the boards. 
Also, new servers aren't the only things waiting for us at the end of this journey :)


======================================


.UPDATE 9:12 PMMost boards are up and running on C20 now. We do have a number of problem boards that need special attention. Those are being transferred one by one, and supervised. Several of those boards have a lot of data and are going slow, our apologies for this.There will be a short scheduled maintenance on C20, to scale the server. This maintenance will start at midnight Pacific Time (3:00 AM Eastern time) and will last approximately 30 minutes.===============================. UPDATE 12:10 PM

The imports to the new server are nearly finished and are expected to be completed by mid-day or sooner.  There are a few boards that have some problems, those are being taken care of individually. The server is being tuned further, and will start to pick up speed as the day goes on.
You may still experience brief interruptions while the new server is being worked on.

We're received a few reports of images not showing. Please check where your images are stored, Photobucket has servers down, and all the ones we've checked are stored in Photobucket. We don't have control over their servers.

If you see anything out of place, please don't hesitate to report it in this topic.

Thank you.

=================================

. UPDATE at 3:06 AM Eastern time on November 4th.

Hello everyone,

The day has been spent transferring boards to the new server. There are tens of thousands of boards being transferred, some have over a decade worth of data. The transfers are not fast, but they have been progressing throughout the day and night. Meanwhile, a few adjustments have been made to the server, it will be tuned further tomorrow.

If your board is not on line yet, it means that it is still transferring. All boards that have fully transferred are on line at this time. As the night goes on, more boards will come on line.

I can't thank you enough for your patience, you have no idea how much this is appreciated, knowing how frustrating this is to all.


================================= 



.UPDATE 6:16 PM Eastern

The maintenance flag will be put up on C20 for a short time while some glitches are being fixed. The maintenance flag is there to avoid activity on the boards while the fix runs. Any activity could cause data corruption, which needs to be avoided.


. UPDATE at 4:10 PM Eastern

The final 1,000 boards are transferring, which is a small fraction of the boards that have transferred so far. 
A few wrinkles are being sorted on the server side.
Once all boards are transferred, we're looking at a huge turn-on of most boards on the new server.

===========================


. UPDATE 12:12 PM EasternHello everyone,I want to start by giving a very short history, in case someone just arrives in this topic, no need to read all the way down.. We experienced an outage of several servers. Since the plan had been to upgrade to brand new servers in a few weeks (which would have caused extended downtime), it was decided to upgrade now while the boards were down, that way you don't have to be down again a few weeks down the road.What is happening during the downtime?A new server has been set up. It is still being tuned, which might cause a few glitches.The boards are transferring from the old servers to the new. This is causing slowness, and may also cause white pages to the boards that are already on the server.Once a number of boards are transferred, they are being turned back on, this is when you see your board on line.What to expect today?The data transfers are still ongoing, and as long as those are not finished, the boards will experience issues, especially slowness and white pages. Some fine-tuning is being done on the new server still, which may also cause the occasional white page while it reboots.At this time, many boards should be up and more will be brought back on line throughout the day.If you see anything out of place on your board, please do let us know.What to expect later?A very steady, fast, stable server. Boards will load better than ever. We hope this will happen soon.Once more, I want to apologise for the extended interruption. We know that the boards are important to you all, and having to miss them for days is tough at best. Our team is working hard to get all boards operational on the new server.Please stay tuned for further updates.===============================. UPDATE at 2:37 AM EasternThe new server is fully up and running. We are currently waiting for the data transfers to finish. Some boards have been brought on line and have been tested.If you see your board up, it might still run slow due to the data transfers, and also due to the fact that the boards need to build up cache again.It has been a very slow process and I want to thank you all once again for the patience you have shown. Once you're all on the new server and up, things should look very good.I see several boards coming up as we speak, I'll confirm which ones are up via a reply in this thread here./. UPDATE 12:30 AM

We have a bit of a delay in bringing boards on line, the data transfers are extremely taxing on the server, making it difficult to navigate the boards that are already back up. Hang in there...


. UPDATE at 9:35 PM Eastern

We are ready to bring boards back on kine. Please know that there will be some glitches while a whole lot of data is still transferring to the server.
We are starting with some individual boards before we open up the gates. If you'd like to be in the the first group, please post here with the following:

1) A link to your board
2) Your old server number if you know it
3) A link to the profile that owns the board (important, because I can look up the board ID that way)

Also, your board will show the old server number when it comes on line, but it will be on the new server. The numbers will be corrected once all boards are transferred. New server is confirmed C20


==========================


.. UPDATE 8:53 PM

We will have a short interruption while the web server are rebooting.



. UPDATE 8:12 PM

My apologies for the long intermission between updates, at least the news is good.  We're very close to bringing the boards on line (pretty much just minutes away). They will not be brought on line all at once, since some are still transferring. Stay tuned./.


.UPDATE on November 2nd at 2:55 PM

The data transfer is progressing as expected. Replication is currently being worked on. Once that is done, our team can start bringing boards back on line. We are getting closer. Thanks again for hanging in there!


=========================

.UPDATE on November 2nd at 10:28 AM

The transfers are still ongoing, and boards are expected to come on line today. Not all boards will be coming on line at the same time, it will be a gradual process.
Remember, you're going to be on a server that will the old server look like a toddler :)
Again, thank you all for so much patience you've shown.
I will update as soon as I have more news, which should be fairly soon.

=====================================



. UPDATE on November 2nd at 2:18 AM

Data transfers from C2 C6 C8 C10 C11 are underway.
C12 transfer has been rescheduled, and C12 boards should be back up at this time.
The data transfers are progressing very slowly and they are not expected to be done by morning yet, sorry :(

Servers that are up: C3 C4 C5 C7 C9 C12 C14 C15 C16 C17
Servers that are not up: C2 C6 C8 C10 C11

What to expect:
- The data transfer is not a quick process. I don't have an exact ETA yet, and I'd hate to make promises we can't keep. I will let you know as soon as I have an ETA though.
- Search will be down after the boards come back on line.

=========================


. UPDATE 6:55 PM Eastern time

The destination server is up and running.
Data is being exported from C6 and C2, so far, everything is going very smooth.
Export from C8 and C10 has started,
C11 and C12 are next.

At this time, if your board is on any of those servers, it will not be on line yet, there is still a lot to do.
I don't have an ETA yet, but it will likely be an overnight process.

Thank you all for your patience while the servers are being updated to make your boards faster and more stable in the very near future.


.UPDATE 4:30 PM Eastern time

Hello everyone,

My apologies in the delay in updating. Here is the latest. As you know, we have a few servers down at this time. Instead of fixing them up,  the downed servers are being put to rest. They will be replaced with new servers that are currently in the process of being set up.

Disadvantage: The downtime will be a bit longer than expected. I don't have an exact ETA yet.

Advantage: Once your boards come back on line, they will be hosted on brand new, powerful, solid servers.

=====================================

Hello everyone,
We have some servers down at this time. The fix needs to happen via the networks operations center, meaning it will take a few hours for the servers to come back up. I am very sorry for this, especially after what we went through a couple weeks ago, but please know that everything is being done to get you back up as soon as possible.

As usual, please stay tuned to this topic for updates. With each update, the title of this topic will show the exact time the update was made.

Thank you all for your patience while this is being sorted.
Last edited by ChrisW on November 22nd, 2016, 12:40 am, edited 23 times in total.
"One can see clearly only with the heart. Anything essential is invisible to the eyes"~~Antoine de Saint-Exupéry~~
Reply

User avatar
QueenFoxy
Joined: February 12th, 2013, 10:44 pm

November 1st, 2016, 5:32 pm

Could I be put on the new server when this is over.  C11 always seems to be the first to go.  It has had lots of trouble in my years at Yuku.
Reply

User avatar
Banglebabe
4
Joined: November 1st, 2016, 5:38 pm

November 1st, 2016, 5:38 pm

Is this why my page is saying "HTTP 500 error
That’s odd... the website can’t display this page
The site may be under maintenance or could have a programming error." I just opened a ticket, so wasn't sure now if it's for this reason. Thanks for any replies.
  
Reply

User avatar
QueenFoxy
Joined: February 12th, 2013, 10:44 pm

November 1st, 2016, 5:55 pm

I opened a ticket too, Banglebabe.  Then I saw this announcement and I knew.....?   Not good news for today.
Reply

User avatar
MH370Admin
Joined: October 17th, 2016, 8:39 am

November 1st, 2016, 6:23 pm

Hi Chris,

Sorry to learn about the server trouble.

If it's any help to you, I encountered this error message about 2 hours ago when I was unable to access my board:

Fatal error: Uncaught exception 'Exception' with message 'Objects returned by SelectQuery::getIterator() must be traversable or implement interface Iterator' in /cg_sites/yuku.com/htdocs/lib/User/User.php:324 Stack trace: #0 /cg_sites/yuku.com/htdocs/lib/User/User.php(324): User::identity_domain() #1 /cg_sites/yuku.com/htdocs/controller/global/global.php(92): User::identity_domain(NULL) #2 /cg_sites/yuku.com/htdocs/index.html(105): controller_global::scope(Array) #3 {main} thrown in /cg_sites/yuku.com/htdocs/lib/User/User.php on line 324

Best regards,
MH370Admin
http://mh370.yuku.com
Reply

User avatar
ChrisW
Joined: January 31st, 2013, 10:58 pm

November 1st, 2016, 6:58 pm

I am waiting for an update from the technical staff and will post the update as soon as I have it.

MH370, thanks, useful info, and appreciated.

QueenFoxy, yes you can have your board moved, just add your link in the new server topic.

Banglebabe, yes, this is why you are seeing a 500 error.
"One can see clearly only with the heart. Anything essential is invisible to the eyes"~~Antoine de Saint-Exupéry~~
Reply

User avatar
bewildered
Joined: March 18th, 2013, 10:16 pm

November 1st, 2016, 7:23 pm

It wasn't me. I wasn't here.

I have an alibi
Reply

User avatar
Joybeetroot
Joined: April 19th, 2014, 2:38 pm

November 1st, 2016, 7:33 pm

LOL at you Bewildered , I am keeping my fingers crossed we will be back on before my bedtime lol
Reply

User avatar
Nottie
Joined: March 2nd, 2013, 8:27 am

November 1st, 2016, 7:40 pm

Bew go to your broom cupboard please
Thanks for keeping us informed ChrisW
Reply

User avatar
ChrisW
Joined: January 31st, 2013, 10:58 pm

November 1st, 2016, 7:50 pm

I am going to get a full update within the next 30 minutes, and will update the lead post asap.

Maybe Bewildered can show us his alibi while we're waiting :)
"One can see clearly only with the heart. Anything essential is invisible to the eyes"~~Antoine de Saint-Exupéry~~
Reply