So we messed up, plain and simple.
The media server was improperly terminated while during a "routine" adjustment of the load balancerWhile not thought to be a big deal initially (We are masters of backups!) We figured we'd just restore the latest backup and be on our way. Soon after it was discovered that the 3 separate backup methods weuse all contained a single point of failure stemming from a permissions issue.
So the good news and the bad news.
So where does this leave us?
We were able to scrape our CDN and recover some of the media, but ~60% of the media uploaded in the past month were lost.This issue only affects uploaded images and videos. Account security, wall posts, friend requests, messages were not affected by this at all.
We are working to notify the users that lost images/videos to give them the opportunity to re-upload their images/videos. This could not have come at a worse time, we are sponsoring the Olive Dell 5k in california, so Iwill be working round the clock (3 hours sleep so far) until anything recoverable is back in place. The backup system has now been updated and contains a monitoring system to ensure it never happens again.
We appreciate your support of the site and truly apologize for this inconvenience.
