Craig Parker Fans (original) (raw)
sindaringoddess @ 03:38 pm: More EZBoard updates. Woo.
Dear ezboard Family,
Good morning and thank you for your patience as we continue to restore your data and return our systems to normal operation.
This will be a long update, as I want to be as thorough as possible and incorporate threads from the many phone calls and e-mail discussions I have had with ezOps, admins and users in the last few days. I hope that by being open and explicit we can give you a better understanding of what has happened, where we are in the restoration process, and when we can all expect to get back to normal.
I've broken all this information into topic areas to make it easier for you to find the topic that you personally care about the most.
When will my board get its data back?
The most difficult and painful thing for me to report is that it will be impossible to restore full data on all boards. We will send out a Global Announcement once all possible restoration procedures have been completed. During this process, you can expect to see some slowness on the boards, but we’ll return to normal speeds as soon as the restoration is complete.
I realize this creates great uncertainty for many of you still waiting for data restoration, and so I want to give you the following, much more detailed description of the data restoration process we are undertaking in our efforts to get the most back we can, on the fastest time frame possible:
Data restoration began the day of the attack with the restoration of eight servers that we could salvage immediately. Sadly, no other intervention was successful at that time.
Since that time, we have been assembling data from three sources: the remaining historical backups of some 8,000 Gold and trial boards from early May, the current data recently written to hard drives since the attack, and data in RAM memory since the attack.
The actual restorations using this assembled data have been underway since Thursday afternoon, and are being done on a server-by-server basis. The process is automated and each server takes approximately 6 hours to restore. We can restore between 8 and 10 servers in parallel. To some of you this may seem like a long time to restore data on a single server. And it is. However, we are using an innovative process to maximize the data recovery and testing it thoroughly to make sure that (a) we don't erase the posts that have been made since the attack and (b) the data is being properly restored.
When this server-by-server process is complete, we have one final source of data that we may be able to use for restoration. With quick thinking and fast action on the part of your technical team we were able to do an emergency backup of active data. To be clear, this is by no means all the board post data, only the active data at that time. We will be restoring this data as a last step in the process.
We are working as fast as possible on the restoration process - believe me, the faster this is completed the happier we all will be.
What happened in the attack?
What we know is that late in the evening on the holiday Monday, May 30 here in the US, a person with significant and specific information attacked our systems. The attacker was successful in erasing historical board posts and the backup systems, which were password protected and on a separate network. They did not erase the posts of boards that were active on the system at the time.
What more can you tell me about the back-ups?
The issue was not the back-up systems themselves. Our back-up systems have been working well and are separate from our production environment. They are not accessible without inside technical information and passwords. The issue was that either through luck or (more likely) specific information, the attacker was able to delete both the historical files and the back-ups.
What are you doing so this doesn’t happen again?
We have put additional safeguards in place to reduce the opportunity for this to happen in the future. We will also be undertaking a highly detailed security and back-up audit in the coming weeks.
You should also know that we have been Alpha testing the new community software for three weeks and in addition to increased security, redundancy and back-up systems, it will have the ability to allow ezOps to back-up their communities (in addition to daily and weekly incremental backups) to their home computers in an automated fashion. For some ezOps this should provide an additional level of safety and peace of mind.
Can't you give us more frequent updates?
We have been updating you as soon as there is new information. Since the attack we have put out five global messages and have had two conference calls with ezOps and admins. In addition, the Customer Service help desk and Support Forums have been very active. On a more personal basis, Steve DeMello and I have been responding to hundreds of e-mails from users, ezOps and admins. As of 7AM PST on Friday we have responded to every e-mail that has been sent directly to the two of us.
If you are looking for ways to get up to date information fast I recommend reading p079.ezboard.com/fezboard...ouncements first. Please read the materials already available before contacting CS -- the response queues are pretty full, as you might imagine.
What can you tell us about the attacker?
I can tell you that we have suspects, we have evidence and we are working with the FBI. I can also tell you that we will be working to have the attacker punished to the full extent of the law.
While we do have evidence, we can always use more, good, factual evidence. We are offering a $5,000 reward to the person or persons that provide us with factual information that leads to the direct arrest and conviction of the attacker. We are not interested in conspiracy theories, finger pointing or childish pranks. We are interested in information such as community postings or blog entries where the attacker is bragging about the attack and other factual, real information. Please e-mail information to legal@ezboard.com. When you e-mail please provide a link to the original material AND a screen capture of the original material or at the very least, copy and paste the original material into your e-mail. Please be smart, ethical, law abiding and honest about the way you find the information your provide to us.
Now the fine print: A reward will be paid to the person or persons that deliver to legal@ezboard.com information that leads to the direct arrest and conviction of the person or persons that attacked ezboard.com earlier this week. Should more than one person provide identical or substantially similar information that is used for the arrest and conviction of the attacker, the reward will be split equitably amongst the providers of that information.
Will you be providing Credits for Gold Boards?
Yes we plan to provide credits. We need to understand the scope of the data restoration before we decide on the amount of the credit. I would anticipate further news on credits to be available in the next week.
What happened to my Community Chest Balance?
The Community Chest balances were deleted as well, but we do have a back up of that data from a few days before the attack. We need to recalculate the actual Community Chest balances by taking the balance as of the date of the backup, adding Community Chest contributions and subtracting payments from Community Chest. We hope you agree that restoring Community Chest data is a second priority to getting the board posts restored
Please don't worry about your Community Chest balance, it will be restored and updated.
So, where do we go from here?
First let me say that we at ezboard are deeply sorry about the attack this week. The person that did this to us probably had a great chuckle when they pressed the send key to initiate the attack. Little did they know that they would affect so many lives in so many ways. I have heard from cancer patients, historians, mothers and schoolteachers about the way that their boards were damaged and were of vital importance to them. What the attacker did was take a great big piece of many peoples lives including the folks here at ezboard. For that, I hope we catch the attacker and return the favor by taking a bit of their time - behind bars.
We are working to restore the boards as fast as we can. We know your trust in ezboard has been deeply shaken. Now we are trying to rebuild that trust, and are realistic that this will be difficult and take time.
We plan to keep messaging you on our progress. Some of you may have already seen data restored as we started the process on Thursday. Please check p079.ezboard.com/fezboard...ouncements for updates.
In closing, we are working to get systems back to normal and believe that they will be back soon. We are fixing the security and backup processes now and will be performing a security and back-up audit in the coming weeks. Thank you for being an ezboard user, and for reading this, and we hope that you will continue to use ezboard. Let’s put our communities and our network back together, and bring the attacker to justice.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
Current Mood: 
bored