We are done with you!

You're right. There is no excuse for the downtime or to having no staff be reachable. We'll be sad to see you go.

I've been sending a variation of the following message to customers, but I should post it here as well:

I truly apologize for the downtime. Over the past several months weve been working to ensure that no single part of the infrastructure is make or break. Weve been successfully chipping away at each corner of this infrastructure.

We did have a single load balancer server that was make or break, however, and we hadnt yet designed an automated solution to this problem. It just so happened a couple of hours ago the people in the datacenter decided to reboot a bunch of machines (including this one) and the routers were not correctly assigning IP addresses to the machines. Were still looking into the reason why the reboots occurred in the first place (likely, operator error). However, because the correct IP address was not assigned to this entry point, this meant this load balancing machine was alive and running, but did not have the correct IP addresses assigned to it and thus it was unreachable.

It goes without saying that this is embarrassing and were going to make our systems more robust to handle human error. This should not happen especially considering the engineering effort weve devoted to making sure this doesnt happen. And if it does happen our automated systems have to be smarter about fixing it. Right now we failed at that, and for that I apologize.

- Wyatt O'DayFounder & CEO of wyDay

Hi again.

I know that you are working hard to create good projects. And I must say that we really like your both products. It is easy to use, fast and secure. But the downtime was a really disappointment for us.

If you say that you will fix the load balancer problem **like yesterday**, we'll continue to use LimeLM.

And we are also thinking to get it run on our servers. I've registered with the same username on limelm so, you can find my email address. Can you please provide me the details about getting limelm run on our servers? (Price, server requirements etc...)

>> "If you say that you will fix the load balancer problem"

Yes, we're working on an automated solution to this problem and other "unknown unknowns". Companies like Netflix solve these types of "unknown unknowns" type situations by introducing random shutdowns into their production servers (they call it Chaos Monkey). We'll likely need to develop and introduce something similar to ensure we can handle problems we don't even know we have.

>> "Can you please provide me the details about getting limelm run on our servers? (Price, server requirements etc...)"

Sure.

Also, FYI, for other customers, the correct way to use TurboActivate is to use TA_IsGenuineEx(). Never use TA_IsGenuine() (at least not until you've read the documentation and understand the ramifications for using it): https://wyday.com/limelm/help/faq/#require-internet

If you use TA_IsGenuineEx() like we describe then minor downtime (planned or unplanned) *will not* effect you. So, follow our instructions and you'll be fine.

[removed personal attack] Wyatt: you have an excellent product. The outage was overnight and very short. Product is well-worth the money. Could have some more transparent documentation (see my thread), but Wyatt doesn't deserve to be attacked. Thanks.