I dropped the ball
In this post I want to let you know what happened to cause Twitpic to go down for roughly 24 hours on Sunday and to let you know what the plan is. I’ll start off by what happened and what went wrong.
Twitpic was going to be down for scheduled maintenance Sunday morning at 12am CST. During this maintenance time I was moving some of Twitpic’s servers around in our data center and getting things ready for the switch to Twitpic’s new hardware platform that is going live sometime this quarter. The move itself went fine, but after the move a hardware failure occurred on one of our main servers that caused Twitpic to run slow and then eventually go down completely. After troubleshooting for hours and realizing I couldn’t fix the server since the issue was due to a hardware failure, I setup a new server to take over.
Twitpic being down that long is not acceptable. I am very sorry and I (@noaheverett) take full responsibility.
My main goal is to make Twitpic stable and to prepare for our growth. For the past few months I’ve been building a scalable hardware platform to run Twitpic on and I will be going live with that this quarter. I’m also taking the appropriate means to get funding into Twitpic. I do realize that me running the service by myself will become a bottle neck to Twitpic and so I am looking to grow Twitpic from just me, to a great team.
I truly am thankful for the support you have given me and for your words of encouragement. I want to pay you back by giving you the best service possible. I really believe we can change the world with this.
January 21st, 2009 at 10:52 am
Thanks for providing a great service and finding a solution quickly!
January 21st, 2009 at 10:53 am
dude, awesome. it really made it obvious during the down-time just HOW valuable and usefull twitpic is. this, should make it quite easy to find funding amongst the THOUSANDS of users that couldn’t post their pics!
glad to have you back dude!
January 21st, 2009 at 10:54 am
I think it’s great you want to let us know what went wrong, but I don’t think it was necessary to write a blog about it, I would like to think that everyone that uses the net knows that servers get full in time and onece in a blue moon something like this will happen. Again thanks.
January 21st, 2009 at 10:55 am
Aw you are too hard on yourself! We love twitpic – its a free service to us and you do your best! Keep it up.
January 21st, 2009 at 10:55 am
I think you did a spectacular job of keeping the users updated, Noah. I think a lot of us can appreciate your transparency through this whole process. Other services aren’t quite as great at that! Thanks for keeping us in the loop.
January 21st, 2009 at 10:55 am
Hey, it says so much that people were stressed about ‘TwitPicTrouble’ – it adds so much to Twitter, and we really missed it :). Good luck with the new platform. We’ll be watching ;).
January 21st, 2009 at 10:56 am
You already do superb work, offer a fantastic service, and do it all by yourself. I’d give yourself a pat on the back for doing as well as you have and wish you every success in the future.
PS: If you’re looking for people for your team in future, let me know!
Mark
January 21st, 2009 at 10:57 am
Great job!!!
Would be cool to see some twitpic pics of the new hardware you are running for us geeks :-)
Cheers
January 21st, 2009 at 10:58 am
hey – no beef, we know you’re doing your best. What more could anyone expect from a free service?
January 21st, 2009 at 10:58 am
Don’t worry about man. You a one-man show trying to maintain a site that has exploded in recent months. It was bound to happen. I always like to say that these types of problems are the good problems. It means your business is growing because you’re doing something right. It’s when you start losing traffic you need to start worrying.
Good to see everything is running smooth again.
January 21st, 2009 at 11:04 am
I demand a 100% refund of all the money I have paid for TwitPic NOW! My business/life/marriage/livelihood/career/social standing are all in shambles because of this outage. I can’t believe your staff of hundreds took so long to fix this problem. Completely unacceptable.
Um, yeah, just kidding. Thanks, Noah, for your transparency, and most of all for TwitPic! I almost used your crosstown rivals, ping.fm, during your outage, but I (and my vast array of followers) somehow managed to survive without any photo postings.
Thanks again.
January 21st, 2009 at 11:04 am
Thanks for all you do. I look forward to all the new developments
January 21st, 2009 at 11:06 am
I want my money back! LOL just kidding, thanks for providing a great service and fixing things to the best of your ability, Love Twitpic!
Keep up the good work!
Tim
January 21st, 2009 at 11:10 am
Only just started using Twitpic and I think you are doing amazingly well running a free service like this. Well done for getting it back up and online so quickly.
January 21st, 2009 at 11:10 am
Nice way of owning up and taking control of the situation. Good luck in all future endeavors.
January 21st, 2009 at 11:11 am
Ditto what all the people said about no worries.
January 21st, 2009 at 11:13 am
you guys rock. keep it up.
January 21st, 2009 at 11:13 am
Thanks for providing this great FREE service. To be honest, when TwitPic has issues, I don’t complain… why? Because it’s a FREE service. We are lucky to have it. Thanks for keeping TwitPic free!
January 21st, 2009 at 11:16 am
Your one man show has been great thus far! Until you get funding (and probably after), I’m sure there plenty of tweeple out there (like me) who would be willing to lend a hand where they could to help you develop the service. Hit me up if there’s anything I can do for you. Keep up the great work!
January 21st, 2009 at 11:16 am
I want a refund, too! LOL!
Thanks for all your hard work on TwitPic. I love the service and fully understand that there will be some growing pains along the way as the service evolves.
January 21st, 2009 at 11:21 am
you’re doing a fantastic job! twitpic is an amazing service! thanks for all that you do.
January 21st, 2009 at 11:27 am
Don’t worry. I have faith that you will ensure that the system is stable and grows strong. I don’t use it much myself but it is an awesome thing, to be sure.
January 21st, 2009 at 11:28 am
Hey, no worries, matey. You rock! :)
January 21st, 2009 at 11:39 am
Keep up the wonderful work, Noah! It’s incredible to know you’re doing this all by yourself.
Do you consider any possibilities for us, the users, to assist you with funding? I’d be glad to do that.
January 21st, 2009 at 11:39 am
Twitpic is great—thanks for getting back up and running so quickly! Looking forward to lunch in February.
January 21st, 2009 at 12:03 pm
Hey, stuff happens. You provide such a great service, there’s grace!
January 21st, 2009 at 12:06 pm
twitpic is amazing, and I’m impressed at how hard you work to get it back up and keep everyone informed.
If you need funding, might I recommend a “donate” link, a’la wikipedia? I really think you’d get a pretty decent amount. I know I’d throw in my ten cents (figure of speech, I’m not that cheap.)
January 21st, 2009 at 12:06 pm
Only just started using TwitPic but already love it, I am sure users would be willing to make a donation for whatever one can afford.
January 21st, 2009 at 12:22 pm
This is simply growing pains. You’re providing a fantastic service and frankly – Twitter should just buy Twitpic like they did with Summize.
Side note – Noah, if you’re interested in monetizing your pages with some display ads just DM me.
January 21st, 2009 at 12:48 pm
I’m very happy that I discovered Twitpic and am impressed with what you are doing.
January 21st, 2009 at 1:04 pm
I bet the stress levels were through the roof during the down time.
Keep up the good work. Twitpic provides a great free service.
January 21st, 2009 at 1:31 pm
Much respect, Noah. And most people don’t even know the miles you made without sleep. Ttown-OKC and back, Ttown-Dallas and back, twice! I’m glad you survived without crashing.
January 21st, 2009 at 1:39 pm
twit happens. No sweat. I still can’t believe there’s only one of you….
God bless!
Cindy
January 21st, 2009 at 3:40 pm
I think you’re too hard on yourself. As cindy said, “Twit happens.”
It’s sometimes good marketing when your audience misses you.
I gotta be honest though. If you’re not part of the twitter team, how can you build a service off of their brand and still get away with it? Did they not TM themselves?
January 21st, 2009 at 5:46 pm
It’s OK, I was taking sunday off so you can too! ;-)
Seriously, great to hear things are rocking again!
Adam in Phoenix.
January 21st, 2009 at 7:13 pm
Anyone can provide great service when things are running well. The mark of exceptional service is how you handle a situation when everything blows up. The communication was helpful and your candor was refreshing. I just started using the service a couple weeks ago. Had your updates not been so well done, I might have walked away and missed such a great service. You have clearly earned your followers, and I’m pleased to be counted as one of them.
January 22nd, 2009 at 12:05 pm
Guess I should have been following you~ then I wouldn’t have had to email you yesterday about twit pic ;) Googled you ~
You’re awesome!! This is one of my favorite things online~ twitter & twitpic!!
January 23rd, 2009 at 12:46 am
Noah,
Thanks for the hard work!!! Seriously!!! And get some rest too!! You deserve it. =)
Jessica
January 23rd, 2009 at 1:25 pm
Dude… it’s okay. I have faith in you!!!
January 23rd, 2009 at 3:51 pm
Noah,
Thank you for for the great update and service. I love using Twitpic with Twitter. I’m sure I’m not the only one that may have taken for granted a service that is free to them, but has come to expect it to always be functioning and accessible.
Keep up the good work and best wishes in getting the funding you need to grow.
David G.
February 10th, 2009 at 1:48 pm
Thank you for your hard work and more than that – thank you for the wonderful, human, natural response, such a delight to read.
February 15th, 2009 at 5:37 am
Can you please take a look at the function to delete pictures? I have followed all instructions and there is no trash can. It’d be greatly appreciated.
March 6th, 2009 at 2:55 pm
I have seen what Twitter can do so I know if you can stay strong and continue with this it will be a great thing!
March 7th, 2009 at 8:56 pm
I’m with #42 James, I cannot delete pictures and I have duplicates coming out my left ear… or something like that.
I also changed my Twitter name, but Twitpic thinks my name is still the old one. I cannot see comments made to my pictures as Twitpic doesn’t seem to know I changed names and sends me to the old nickname with zero pictures, but 2 comments.
I realize you’re working as hard as you can, and I truly appreciate all of your time and efforts.
Thanks!!
March 11th, 2009 at 8:16 pm
Would you be able to allow cellphone video posts the same way you do pictures?
March 31st, 2009 at 11:15 pm
just keep the joint rolling…..oopsss the ball….peace…
April 4th, 2009 at 1:41 pm
I LOVE HAMSTERS!!!!!!
April 24th, 2009 at 6:48 pm
I think you should write what you feel
April 24th, 2009 at 8:28 pm
Oh my good God! Has hell frozen over? Are pigs flying? Did I just read a post that was brutally honest? Honest about someone’s mistakes? A post clearly written by a real human, with conscience and thought-process? A business post, pushing negative publicity about the business, because that’s how they run business? because the bottom line is clearly not the most important thing in ur business ethic? A post not muffled or edited by some corporate, greasy haired fatcat, in order 2 save face @ all costs?
I could go on & on & on. You f@#$ing rawk dood! S$#% Happens. We’re human. Live & Learn. Bravo! Kudos! Very refreshing!
Cheers,
J. Albert Bowden II
May 27th, 2009 at 10:59 am
Hi Noah-
I hope all is well in N.C.
Tulsa misses you, and so do I.
Would you consider coming out for D-Fest? You can stay with me.
CYA
Nate