I dropped the ball

In this post I want to let you know what happened to cause Twitpic to go down for roughly 24 hours on Sunday and to let you know what the plan is. I’ll start off by what happened and what went wrong.

Twitpic was going to be down for scheduled maintenance Sunday morning at 12am CST. During this maintenance time I was moving some of Twitpic’s servers around in our data center and getting things ready for the switch to Twitpic’s new hardware platform that is going live sometime this quarter. The move itself went fine, but after the move a hardware failure occurred on one of our main servers that caused Twitpic to run slow and then eventually go down completely. After troubleshooting for hours and realizing I couldn’t fix the server since the issue was due to a hardware failure, I setup a new server to take over.

Twitpic being down that long is not acceptable. I am very sorry and I (@noaheverett) take full responsibility.

My main goal is to make Twitpic stable and to prepare for our growth. For the past few months I’ve been building a scalable hardware platform to run Twitpic on and I will be going live with that this quarter. I’m also taking the appropriate means to get funding into Twitpic. I do realize that me running the service by myself will become a bottle neck to Twitpic and so I am looking to grow Twitpic from just me, to a great team.

I truly am thankful for the support you have given me and for your words of encouragement. I want to pay you back by giving you the best service possible. I really believe we can change the world with this.

51 thoughts on “I dropped the ball

  1. dude, awesome. it really made it obvious during the down-time just HOW valuable and usefull twitpic is. this, should make it quite easy to find funding amongst the THOUSANDS of users that couldn’t post their pics!

    glad to have you back dude!

  2. I think it’s great you want to let us know what went wrong, but I don’t think it was necessary to write a blog about it, I would like to think that everyone that uses the net knows that servers get full in time and onece in a blue moon something like this will happen. Again thanks.

  3. I think you did a spectacular job of keeping the users updated, Noah. I think a lot of us can appreciate your transparency through this whole process. Other services aren’t quite as great at that! Thanks for keeping us in the loop.

  4. Hey, it says so much that people were stressed about ‘TwitPicTrouble’ – it adds so much to Twitter, and we really missed it :). Good luck with the new platform. We’ll be watching ;).

  5. You already do superb work, offer a fantastic service, and do it all by yourself. I’d give yourself a pat on the back for doing as well as you have and wish you every success in the future.

    PS: If you’re looking for people for your team in future, let me know!

    Mark

  6. Don’t worry about man. You a one-man show trying to maintain a site that has exploded in recent months. It was bound to happen. I always like to say that these types of problems are the good problems. It means your business is growing because you’re doing something right. It’s when you start losing traffic you need to start worrying.

    Good to see everything is running smooth again.

  7. I demand a 100% refund of all the money I have paid for TwitPic NOW! My business/life/marriage/livelihood/career/social standing are all in shambles because of this outage. I can’t believe your staff of hundreds took so long to fix this problem. Completely unacceptable.

    Um, yeah, just kidding. Thanks, Noah, for your transparency, and most of all for TwitPic! I almost used your crosstown rivals, ping.fm, during your outage, but I (and my vast array of followers) somehow managed to survive without any photo postings.

    Thanks again.

  8. Thanks for providing this great FREE service. To be honest, when TwitPic has issues, I don’t complain… why? Because it’s a FREE service. We are lucky to have it. Thanks for keeping TwitPic free!

  9. Your one man show has been great thus far! Until you get funding (and probably after), I’m sure there plenty of tweeple out there (like me) who would be willing to lend a hand where they could to help you develop the service. Hit me up if there’s anything I can do for you. Keep up the great work!

  10. I want a refund, too! LOL!

    Thanks for all your hard work on TwitPic. I love the service and fully understand that there will be some growing pains along the way as the service evolves.

  11. Keep up the wonderful work, Noah! It’s incredible to know you’re doing this all by yourself.

    Do you consider any possibilities for us, the users, to assist you with funding? I’d be glad to do that.

  12. twitpic is amazing, and I’m impressed at how hard you work to get it back up and keep everyone informed.
    If you need funding, might I recommend a “donate” link, a’la wikipedia? I really think you’d get a pretty decent amount. I know I’d throw in my ten cents (figure of speech, I’m not that cheap.)

  13. This is simply growing pains. You’re providing a fantastic service and frankly – Twitter should just buy Twitpic like they did with Summize.

    Side note – Noah, if you’re interested in monetizing your pages with some display ads just DM me.

  14. I bet the stress levels were through the roof during the down time.

    Keep up the good work. Twitpic provides a great free service.

  15. Much respect, Noah. And most people don’t even know the miles you made without sleep. Ttown-OKC and back, Ttown-Dallas and back, twice! I’m glad you survived without crashing.

  16. I think you’re too hard on yourself. As cindy said, “Twit happens.”
    It’s sometimes good marketing when your audience misses you.
    I gotta be honest though. If you’re not part of the twitter team, how can you build a service off of their brand and still get away with it? Did they not TM themselves?

  17. Anyone can provide great service when things are running well. The mark of exceptional service is how you handle a situation when everything blows up. The communication was helpful and your candor was refreshing. I just started using the service a couple weeks ago. Had your updates not been so well done, I might have walked away and missed such a great service. You have clearly earned your followers, and I’m pleased to be counted as one of them.

  18. Guess I should have been following you~ then I wouldn’t have had to email you yesterday about twit pic ;) Googled you ~

    You’re awesome!! This is one of my favorite things online~ twitter & twitpic!!

  19. Noah,

    Thank you for for the great update and service. I love using Twitpic with Twitter. I’m sure I’m not the only one that may have taken for granted a service that is free to them, but has come to expect it to always be functioning and accessible.

    Keep up the good work and best wishes in getting the funding you need to grow.

    David G.

  20. Can you please take a look at the function to delete pictures? I have followed all instructions and there is no trash can. It’d be greatly appreciated.

  21. I’m with #42 James, I cannot delete pictures and I have duplicates coming out my left ear… or something like that.

    I also changed my Twitter name, but Twitpic thinks my name is still the old one. I cannot see comments made to my pictures as Twitpic doesn’t seem to know I changed names and sends me to the old nickname with zero pictures, but 2 comments.

    I realize you’re working as hard as you can, and I truly appreciate all of your time and efforts.

    Thanks!!

  22. Oh my good God! Has hell frozen over? Are pigs flying? Did I just read a post that was brutally honest? Honest about someone’s mistakes? A post clearly written by a real human, with conscience and thought-process? A business post, pushing negative publicity about the business, because that’s how they run business? because the bottom line is clearly not the most important thing in ur business ethic? A post not muffled or edited by some corporate, greasy haired fatcat, in order 2 save face @ all costs?
    I could go on & on & on. You f@#$ing rawk dood! S$#% Happens. We’re human. Live & Learn. Bravo! Kudos! Very refreshing!

    Cheers,
    J. Albert Bowden II

  23. Hi Noah-

    I hope all is well in N.C.

    Tulsa misses you, and so do I.

    Would you consider coming out for D-Fest? You can stay with me.

    CYA

    Nate

  24. Pingback: I dropped the ball | Media Recorder

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>