We have technicians being prompted to re-enter their product keys, and they do so with no success. We are also unable to create product keys through our web app. Is the LimeLM server online and operational? I would like to get a quick status before getting our Dev team involved.
Thanks!
Additional information: Here is the error message returned when attempting to verify a key (the same error message is returned when trying to activate a key):
jsonLimeLMApi({"stat":"fail","code":164,"message":"An API key must be used by a single device."})
Here is the calling URL: https://wyday.com/limelm/api/rest/?method=limelm.pkey.getID&format=json&api_key=[valid API key]&version_id=1099&pkey=[valid key]
RetroTorqueWe are also experiencing issues with API currently, and is affecting some parts of our software from working (generating new product keys, updating etc.). This seems to have been the case for at least the past 6 hours.
RetroTorqueSeriously what the fuck is going on, been down for a couple of hours and this genuine made no attempt to fix this.
We have hundreds of business clients and they are all relying on this feature todo their every day works, we trusted this service until now, after reading all the comments I believe Wyatt or whatever the name is, is just a fucking clown.
RetrotTorqueSorry for the lack of information in my previous post, I can confirm that we are seeing the same error message as described by bsyvertson in the first post in this thread.
Not sure if there is anything we can do to bypass this issue? Please advise, thanks.
The server is working fine. Its up and running and accepting requests (and has been).
The error youre getting is covered here: https://wyday.com/limelm/help/api/limelm.pkey.generate/
Namely:
164: An API key must be used by a single device.You can only use an API key from a single device. If you need to use the web API from multiple servers then create another user. You can also temporarily reset this message by logging into the account for which the API key is created and resetting the last used IP address on your settings page.
This is a security precaution. We were seeing many LimeLM accounts having web API calls being made from multiple devices (some even looked like they were being made from client-side JS or from the apps themselves).
Long story short, we go above and beyond to make sure your data is not compromised by ensuring your account is being used by you and you alone.
To sum up, for customers using the web API from a single server (as designed and warned about since day one) they will see absolutely no change. For customers that have integrated the web API directly in their app, they will need to change their web API key and use the web API correctly.
Using the web API directly in your app (or in client-side JS) is dangerous. It gives every single user of your app direct access over your entire LimeLM account. Its the equivalent of giving everyone of your customers your username and password. Hence the abrupt improvement of security. Weve warned about this from day one. Its only recently weve made this enforcement change. And we made it to protect you.
TurboActivate, TurboFloat, TurboFloat Server are all completely unaffected by this improvement in security.
----
Edit:
We've rolled back this change for the next 3 business days. This means any IP address with your API key can access your account.
On Monday, July 2nd, 2018 at 8:00am Eastern Time (U.S.) we'll be re-enabling this security protection.
Please fix any usage of your web API between now and then. This is a critical and necessary security feature.
----
Edit #2 (2019): For a full breakdown of this "issue" and our choices, see this blog post: https://wyday.com/blog/2019/when-in-conflict-security-supersedes-usability/
Andy3Hi Wyatt,
we are currently having a similar issue.
Actually, we have the web api key used in two servers.
One is a PHP script that will generate a license after a sale over fastspring.
The other one is a PHP script (on another server) we use internally - people request trials on our website and we have a custom backend where we approve or deny trials.
It has always worked but today it's not working anymore. Are these things related? Did you do any changes recently? If so and if this is by design, is there a way for us to enable a second web api key for the second, trusted, machine?
Thank you!
Hey Andy,
Yep, as covered in the error code, simply create another user and use that API key on the separate machine. Or ensure all your web API requests go through a single machine. Whichever is easier / cheaper for your to implement.
YepWyatt,
Making breaking changes to an API without warning is bad form. You say this was documented "since day one", but I just had a look at your API docs on archive.org and that seems to be incorrect (https://web.archive.org/web/20160827124501/https://wyday.com/limelm/help/api/limelm.pkey.generate/) - it certainly wasn't documented back then.
Plenty of your customers are using this API in perfectly safe ways, and don't appreciate having functionality broken without warning.
I'm sorry, I was unclear. The error code is new (you're absolutely correct).
The fact that API keys should *never* be used from inside apps directly or from client-side JS *has* been documented from day one. We even mention it on the settings page directly above the web API key (and a variation of that message has existed since day one).
This is a breaking change for customers whose accounts are at risk. Namely, whose web API key is likely in the hands of a 3rd party and thus all of their data is at risk. Our change immediately stops the chance of *your* data leaking out to people who shouldn't have it.
It's part of our responsibility to ensure your data is safe. We don't take these types of changes lightly. And we knew this change in particular would upset a handful of people.
Andy3Sorry for the additional post (i don't like to spam 🙂 ) but i wanted to let you guys know that creating the second user obviously fixed it. You guys rock! Keep it up!
YepHi Wyatt,
It also breaks (without warning) clients who were using the functionality in perfectly safe ways, such as, for example, internal desktop admin apps that share an API key, or CLI tools that share an API key, or internal scripts that share an API key. These clients are not at risk, but they HAVE had their functionality broken without warning.
Yes, correct, that use-case will be broken. However it's a bad security practice to share the same access key (whether it's a user/pass pair or an API key) among multiple parties in an organization. Each user that needs access should have a separate LimeLM account (so you can tightly control who has access and who should no longer have access).
Also, and I know it might seem silly, but sharing the API key among multiple users in your organization *is* a violation of the Terms of Service ("4. Your login may only be used by one person a single login shared by multiple people is not permitted. You may create separate logins for as many people as your plan allows."). That has been in the ToS since day-1.
Long story short, I know this will make more work for you. But these are legitimate security improvements. We did not make these changes lightly. Nor were we particularly eager to make them (we had to devote resources to get this right, give customers to flexibility to fix their mistakes by adding "reset" functionality in the settings page, and testing extensively).
YepHi Wyatt,
Fair enough. Thanks for addressing our concerns. We obviously would have preferred some more warning, but otherwise I can see why you made the changes you did, etc.
MattHi Wyatt,Our license generation scripts in FastSpring have stopped working as of today. Is this part of the same problem? The code is essentially just your boilerplate from the FS example. Urgent issue!Matt
Hey Matt,
Reset the last used API key on your settings page. Things will work immediately on FastSpring. And if you use the API key elsewhere, create separate users and use a separate API key for each separate server you're using the API key on.
MattWyatt wrote:> Hey Matt,> > Reset the last used API key on your settings page. Things will work> immediately on FastSpring. And if you use the API key elsewhere, create> separate users and use a separate API key for each separate server you're> using the API key on.
This is completely a completely bonkers change to make without advance warning Wyatt!! I have no idea and no control over how many different IP addresses FastSpring might use to access these scripts. I host and run validation scripts on my website that FS accesses and also a script hosted by FS. Please remove this new check of yours from my account as soon as possible I don't have time to sort this out today and I have lost sales already.
MattWyatt wrote:> Hey Matt,> > Reset the last used API key on your settings page. Things will work> immediately on FastSpring. And if you use the API key elsewhere, create> separate users and use a separate API key for each separate server you're> using the API key on.
And no, it won't work immediately because as soon as FS calls a validation script on my page (which calls WyDay) it will presumably lock my account today. I don't have time today to change all of my validation scripts or indeed to ensure that FS only calls your API from a single IP address. Please remove this check for my account and send emails out well in advance of this sort of breaking change in the future. This is crazy.
diogovitzoThis change without any warning is absolutely ridiculous. Was trying to find out why the f our support agents were receiving 100+ tickets today from users claiming they didn't receive their keys, just to discover changes were made to LimeLM without any warning. This sucks.
diogovitzoAlso, this single IP restriction is stupid to say the least.
One of our products operate on multiple payment processors (Paddle and FastSpring), and they obviously have different IPs. How is this supposed to work out?
I'm in the same boat as others who use FastSpring. I woke up this morning to a host of failed orders and unhappy customers looking for their purchased license keys. Now I'm madly trying to make things right for those customers while I also work with FastSpring support to correct the affected orders so that they're in a complete and whole state (for one thing, they need the license keys associated with them for subscription re-bill purposes).
I've reset the client IP address, but I'm really concerned that FastSpring may issue requests from what appear to be multiple public IP addresses and I'm just going to find myself in this position again.
I agree with others that breaking API changes like this should NEVER be introduced. I understand your point about security, but this is the model that YOU GUYS gave us up until now, so that's not on us. This should have been an opt-in feature and not forced on everyone without knowing how clients are using the API.
Wyatt, you guys have a good product, but this is a bad move. You've put your own customers in a position where they're dealing with their own customer satisfaction issues.
I would very much appreciate it if you'd make this optional for us. I for one would like to disable it until I know for a fact that FastSpring will be issuing requests from a single IP address. Otherwise I have zero confidence that the FastSpring/LimeLM integration will continue to work reliably, and frankly that gives me major indigestion...
MattAgree completely with SCWells. Remove this "feature" immediately for now. Think about it, consult with your users and give us an opt-in and plenty of notice (separate topic perhaps, but I never receive emails from WyDay about API changes). Nowhere in the docs I used to integrate FS did it talk about a need for a separate API key for each IP address, although that might have changed now. More importantly, we have no control over how many different IPs services such as FastSpring may use when calling these scripts. Also not all of us are even on static IP websites (we weren't until recently), so when the IP changes we are screwed for 24 hours. It sounds like FastSpring support are also working overtime today to correct these orders.
We've rolled back this change for the next 3 business days. This means any IP address with your API key can access your account.
On Monday, July 2nd, 2018 at 8:00am Eastern Time (U.S.) we'll be re-enabling this security protection.
I'll edit my first response in this post to make this clear.
Please fix any usage of your web API between now and then. This is a critical and necessary security feature.
diogovitzoFurthermore, we use CloudFlare service on all our websites, and CloudFlare always uses at least 2 IPs. Please let us know if you'll be reverting this change or if I should start looking for another license management platform. Thanks.
(if you really want to proceed with this, adding a IP whitelist might help, but SINGLE IP will never work)
MattHave you confirmed with FastSpring that they use only a single, non-changing IP address to run these license generator scripts? If not, and you enforce this change, then we will have to go elsewhere for our licensing as WyDay will become unusable for us.
Please read my responses above. We've rolled it back temporarily. (Until July 2nd -- in which it will be re-enabled).
The solution: read the error description. Or my previous posts.
In short:
1. Don't use the web API key in client-side JS.
2. Don't use the web API key in your app.
3. Every separate server that uses the web API *must* use a separate web API key. (How you do this is described above, in the error description, and on the settings page -- please read any one of those)
4. Using cloudflare to protect a service does not change *the server's* IP address. It simply changes how customers accessing your site see as an IP address. In other words, this is completely unrelated.
FYI, I've also built out a self-service console for users to perform offline activations, manage license key activations, etc., that is hosted at Heroku. I'm setting that up to use a separate user/API key now so it doesn't conflict with FastSpring for order fulfillment, but now I'm even more concerned about variable client IP addresses because of the nature of such a transient application.
Bottom line, we really need to be able to disable this new security feature if it interferes with our ability to do business. I also agree that the current model of a single IP address per-API key, per-24 hour period is untenable for modern cloud computing. Some other protection method is going to be required if you want to pursue such a secondary security mechanism because this one won't work.
MattI've raised a support ticket with FastSpring to find out if they can guarantee that the IP of the calling server won't change, and that there is a single public-facing IP.Wyatt - what happens if an IP changes periodically? How does the "24 period" work? Presumably the IP is allowed to change occasionally without loss of access, so long as there has been at least 24 hours since the last change?
>> "Presumably the IP is allowed to change occasionally without loss of access, so long as there has been at least 24 hours since the last change?"
Correct. Or you can "reset the timer" on your settings page (directly under the API key).
MattWyatt wrote:> Or you can "reset the timer" on your settings page (directly under> the API key).
I don't see this option. Is it a button? What does it say?
I have heard from FastSpring, who have confirmed a single IP address that is unlikely to change, so there is some good news at least.An email in advance next time please Wyatt.
On your settings page: https://wyday.com/limelm/settings/
The message will look like:
"This API key can only be used from a single IP address per-24 hours. If you wish to use the web API key from multiple IP addresses, create separate LimeLM users. This API key was last used from [IP Address of Server] within the last 24 hours. Reset last-used IP address to use the API key from a different IP."
And clicking "Reset last-used IP address" will do just that.
Wyatt, if we find that our external integrations with online stores, etc., do end up resulting in multiple client IP addresses within the same 24-hour period, how will that affect this change? Basically we'll have two components of our total solution that don't/can't work together and no ability to control it. As you've certainly ascertained by now, that's my big angst here. Can you share your thoughts on that because it seems possible, perhaps even likely?
MattWyatt wrote:> On your settings page: https://wyday.com/limelm/settings/> > The message will look like:> > "This API key can only be used from a single IP address per-24 hours.> If you wish to use the web API key from multiple IP addresses, create> separate LimeLM users. This API key was last used from [IP Address of> Server] within the last 24 hours. Reset last-used IP address to use the API> key from a different IP."> > And clicking "Reset last-used IP address" will do just that.
No, I don't have that message.
>> "No, I don't have that message."
If you've recently reset the API key IP or you've recently changed the API key you won't see that message. Also, you'll only see it from the account from which the API key belongs. So, if you have multiple user accounts for the same LimeLM account, log into the correct account.
>> "Wyatt, if we find that our external integrations with online stores, etc., do end up resulting in multiple client IP addresses within the same 24-hour period, how will that affect this change?"
The "major" external service that most people use is FastSpring, and they have a constant IP address.
If you're hosting services in the cloud, make sure you lock a specific IP address to your service. Different cloud providers have different ways of doing this. Most do it by default. Others you have to buy specific services from them. It depends on the service.
And if you're using the web API from more than one server, create separate users (and thus separate web API keys). See above, the error message, and the settings page, each explaining how to do this.
Wyatt wrote:> The "major" external service that most people use is FastSpring, and they> have a constant IP address.
Wyatt, do you mind sharing your source for this information? I've asked FastSpring whether this is the case, and it looks like at least one other person here has as well. I've yet to receive confirmation, though. I'd certainly appreciate it if you could put that part of the concern to rest authoritatively.
Previous interactions with FastSpring. It's an implementation detail on their side of things. I have to take their word for it.
You can contact them an verify it yourself.
It sounds like at least two of us (Matt and myself) have requests open with FastSpring to verify whether or not their servers which issue license generator requests all present as a single public IP address. I'll be happy to post what I hear back from them here. Matt, if you hear back first, would you mind doing the same?
Regardless of how we got here and where we are, we have an issue that needs to be addressed. Thank you for rolling back the current changes. However, 3 business days isn't realistic in any real-world development environment, especially if it involves publishing to the App/Play stores. And, with a holiday coming up, development teams are lean. Any possibility for extending the rollback for two weeks?
No. This is a security fix and its time sensitive. Our rollback was really a partial rollback. For customers that weve determined are using their web API key directly in their app weve kept the security improvement in place. Well continue to monitor web API usage and attempt to detect poor usage.
This problem doesnt effect TurboActivate, TurboFloat, or the TurboFloat Server (as stated above). This means theres nothing to push out to App stores.
If youre using the web API key directly in your app, you must change your web API key ASAP.
Putting the web API key in your app or in client-side JS is equivalent to dumping all of your customers information directly on the web in plain text.
So, again, if youre using the web API key in your app change your API key right now. Dont wait one second longer.
YepSo if I'm reading things correctly, you've now done a "bait and switch" on the pricing model.
Someone reading your website would see the claim "Users are your employees at your company that will be logging in and using LimeLM" and think they are safe to purchase a given plan, only to find out that they then need an additional "user" or two for their payment processing web service (and another "user" for their staging environment), and then another "user" for their reporting scripts that run on a separate server, and maybe even another few "users" for their tablet and phone when they access the API from their custom admin/management app on the road.
I'm sorry, Wyatt, but you've either not thought this through, or you're trying to screw your customers.
This is not a financial decision. Nor is it a bait and switch. Like I've said multiple times in this thread (and as the error code descriptions say), this is a security measure.
As much as it's *your* job to protect your customer's data, it's our job to protect your customer's data too. And that's what this whole thing is about.
After Monday I'll write a blog post explaining this decision and why it was both correct in it's timing and execution.
If you need more API keys to fit your back-end needs, shoot us an email at support@wyday.com and we'll put you on a custom plan with more allowed user accounts.
MattSCWells72 wrote:> It sounds like at least two of us (Matt and myself) have requests open with> FastSpring to verify whether or not their servers which issue license> generator requests all present as a single public IP address. I'll be happy> to post what I hear back from them here. Matt, if you hear back first,> would you mind doing the same?
The response that I got from FastSpring was that they do have a single public-facing IP for the server that calls these scripts, and that it is "very unlikely" to change. So that, at least, is reassuring. I still think this situation should have been handled better but I'll leave it there.
Matt wrote:> The response that I got from FastSpring was that they do have a single public-facing> IP for the server that calls these scripts, and that it is "very unlikely"> to change. So that, at least, is reassuring. I still think this situation should have> been handled better but I'll leave it there.
Thanks, Matt. I still haven't gotten a response, but my inquiry was munged in with the other support case on the overall issue that ended up being this API change, so it may have been missed. That's good to know. It's not 100% definitive, but at least it's relatively positive about there not being an unexpected change that would break us.
Wyatt wrote:> As much as it's *your* job to protect your customer's data, it's our job to> protect your customer's data too. And that's what this whole thing is> about.
Wyatt, I don't think anyone would ever disagree that customer data security is absolutely critical, and even more so now that we have things like GDPR that can carry hefty fines if data is misused/compromised. I don't think that's the real source of consternation in this thread, though.
At the risk of sounding like a broken record, I think the two big issues here are:
1) The manner in which this change was rolled out, seemingly without regard to its potential impact on your customers' ability to do business without interruption.
2) The specific nature of the implementation, based on a single client IP address per-API key, per-24 hour period which breaks completely legitimate and secure integrations, especially in a modern cloud-based environment.
You have a customer above who has clearly stated that even with all due diligence, it will likely be impossible to update existing applications that are broken by this change before Monday due to external review processes that are not under the customer's control, and yet the response is that the change will still be rolled out to that customer on Monday no matter what. You've basically just told a customer that you're totally fine breaking their up-until-Tuesday legitimate usage of your product.
Perhaps this is already going on behind the scenes, but I think a leniency period for these specific types of situations is completely reasonable, and based on what you've already said in this thread, it seems that you're able to enable/disable this new behavior on a per-tenant basis, so it should be technically possible to support a staged rollout.
I just don't understand why there doesn't seem to be more consideration for providing a smooth transition to this new model for existing customers. And I apologize for my criticism if that's already happening privately, but it's just frustrating to see fellow customers' angst about how to deal with this unexpected and disruptive change being seemingly dismissed.
My $0.02 as someone who was affected by this change, though not nearly as badly as others here...
Hey Scott,
I got your email as well, I haven't had the time to respond at length yet, but since you cover a lot of the same ground here, I'll just respond to what you've written here.
>> "1) The manner in which this change was rolled out, seemingly without regard to its potential impact on your customers' ability to do business without interruption."
I will address this with more details on the decision making process in a blog post next week. But, yes, I made this decision with clear eyes about the potential impact. It was the equivalent to pressing a big red "stop" button on the factory floor.
Details on the decision will follow. But yes, all the assertions to contrary, this was a deliberate decision. And even the "rollback" was a choice that was made only after we did the work of manually black-listing potentially compromised web API keys.
>> "2) The specific nature of the implementation, based on a single client IP address per-API key, per-24 hour period which breaks completely legitimate and secure integrations, especially in a modern cloud-based environment."
Yes, your completely legitimate use of the web API was broken by my decision. I'm sorry about that.
>> "You have a customer above who has clearly stated that even with all due diligence, it will likely be impossible to update existing applications that are broken by this change before Monday due to external review processes that are not under the customer's control"
The web API key should never be in a place where it's embedded in an app that needs to be reviewed by a third party (app stores, whatever). The fact that this customer is doing this means one or both of the following:
1. They're using the web API key in their apps that go to their customers.
2. They're using the web API key in "private" app that go to their manager's and other tech-support people in their organization.
If it's #1 then yes, once we flip the switch (again) for all customers then any web API calls in their app will break. This is a good thing. I've said this a dozen times in this thread, and it's all over our error codes and documentation, but I'll state it once again:
===> Using the web API key in an app that the end-user runs on their device is equivalent to publishing your username and password and all of your customers data in plaintext on the web. It's a dumb decision that we've warned about since day 1.
If it's #2, then this is breaking our terms of service ("Your login may only be used by one person a single login shared by multiple people is not permitted. You may create separate logins for as many people as your plan allows."). They should simply create new users and have them manage the account through LimeLM.
Either case, yes, my decision for us to create, and then "push the red button" to breaks those 2 cases was the correct one.
>> "You've basically just told a customer that you're totally fine breaking their up-until-Tuesday legitimate usage of your product."
If it falls into either one of the above 2 categories, then it's not a legitimate use of the web API key.
Your usage does not fall into either one of those cases. Unfortunately you, and small handful of other well-behaving customers, were "collateral damage" in our effort to fix egregiously insecure customers.
>> "and based on what you've already said in this thread, it seems that you're able to enable/disable this new behavior on a per-tenant basis, so it should be technically possible to support a staged rollout."
No, not easily. It required hacks which were expensive to create and would be monumentally expensive to maintain.
Thanks, Wyatt. That explanation does help to provide significant context for the decision and a nice breakdown of the types of affected usages of the API.
Being in the group that was using the product in a legitimate fashion but was nonetheless affected, I still think that some type of less binary/disruptive rollout should have occurred, or minimally you should have communicated with folks who fell into that group beforehand to help them to make adjustments prior to rollout to avoid a disruption in ongoing business. That part isn't really up for debate in my opinion. You obviously did enough analysis beforehand to know how to categorize the various usages, so you should have been able to anticipate those who would definitely be affected but weren't misusing the system. It's the apparent lack of empathy for those folks (myself included) that bothers me. This was very much a forgiveness-not-permission strategy, and that's just not acceptable when you're talking about your customers' ability to do business.
I would also like to ask that you be more open to other ways to solve the same problem that don't cause legitimate usages of your product to fail because they're being hosted in IaaS environments using multiple availability zones, etc. When you're talking about software that must be part of an end-to-end mission critical business process such as product sales, high-availability is critical. Your servers are certainly configured as such, as are FastSpring's and other online storefronts'. If we create intermediate service components that use your APIs, those should also be configured for HA so as not to become the proverbial weakest link. In many cases such an HA configuration will result in multiple regional deployments in a geographic cluster, each with its own public IP address. My guess is that your response to that is to say that each availability zone/geographic region should have its own API key, but because your API keys are inherently coupled to users, that gets into the whole discussion earlier in this thread about limits on the number of users available to each account. This becomes even more muddy if you have a true end users of the application. You're effectively co-opting the notion of an end user for another purpose. Again, there are other ways to accomplish the same goal that avoid these types of issues.
Thanks again for the detailed response. It's by far the most informative (and least defensive) that I've seen on this thread. I still don't agree with the manner in which this change has been (and continues to be) rolled out, but I at least understand the context of the change better now.
StephenWelshI completely agree with this statement. What an amateur approach to running a service, zero warning, 3 day 'emergency fix time' right in the middle of holiday season too. This is not the first time there have been breaking changes without proper warning (disabling HTTP from TurboActivate). We all appreciate the need to secure services and customer data, but we also require adequate communication to mange service changes in a controlled manner. This really does test your customers patience, mine is now very thin... A real shame as you have a nice product.
SCWells72 wrote:> Wyatt wrote:> > As much as it's *your* job to protect your customer's data, it's our job to> > protect your customer's data too. And that's what this whole thing is> > about.> > Wyatt, I don't think anyone would ever disagree that customer data security is> absolutely critical, and even more so now that we have things like GDPR that can> carry hefty fines if data is misused/compromised. I don't think that's the real> source of consternation in this thread, though.> > At the risk of sounding like a broken record, I think the two big issues here are:> > 1) The manner in which this change was rolled out, seemingly without regard to its> potential impact on your customers' ability to do business without interruption.> > 2) The specific nature of the implementation, based on a single client IP address> per-API key, per-24 hour period which breaks completely legitimate and secure> integrations, especially in a modern cloud-based environment.> > You have a customer above who has clearly stated that even with all due diligence, it> will likely be impossible to update existing applications that are broken by this> change before Monday due to external review processes that are not under the> customer's control, and yet the response is that the change will still be rolled out> to that customer on Monday no matter what. You've basically just told a customer that> you're totally fine breaking their up-until-Tuesday legitimate usage of your product.> > Perhaps this is already going on behind the scenes, but I think a leniency period for> these specific types of situations is completely reasonable, and based on what you've> already said in this thread, it seems that you're able to enable/disable this new> behavior on a per-tenant basis, so it should be technically possible to support a> staged rollout.> > I just don't understand why there doesn't seem to be more consideration for providing> a smooth transition to this new model for existing customers. And I apologize for my> criticism if that's already happening privately, but it's just frustrating to see> fellow customers' angst about how to deal with this unexpected and disruptive change> being seemingly dismissed.> > My $0.02 as someone who was affected by this change, though not nearly as badly as> others here...
>> "This is not the first time there have been breaking changes without proper warning (disabling HTTP from TurboActivate)"
That's not true.
Old versions of TurboActivate that *only* talk over HTTP (v 3.4.6 and older) still work. We did nothing to deprecate those old versions. In fact, our infrastructure is still 100% compatible with all old versions of TurboActivate (even closed-beta TurboActivate 1.0 versions from a decade ago can still activate and deactivate with LimeLM).
The fact that newer versions (3.4.7 +) require HTTPS is a good thing. Requiring HTTPS on newer versions fixed many "bugs" (i.e. ISPs spying on and injecting garbage into communication to/from TurboActivate).
So, if you're using an old HTTP version of TurboActivate, and it's not working, point the finger at your customer's ISP and/or network admins. TurboActivate and LimeLM don't exist in a vacuum. We work our asses off to maintain backwards compatibility and fix other people's bugs so things "just work". But there are some things that are unfixable. For example, if an ISP ruins HTTP communication to/from an old version of TurboActivate and LimeLM, we can't magically fix the ISP. The only thing we can do is remove the ISP's ability to ruin that communication. Hence our decision to force HTTPS usage for TA 3.4.7 and newer.
But this is all way off-topic.
Long story short, we bend over backwards to maintain backwards compatibility. We didn't break compatibility here.
>> "We all appreciate the need to secure services and customer data, but we also require adequate communication to mange service changes in a controlled manner."
I agree. And if this were a normal circumstance that's exactly what we would've done in the form of a blog post here with adequate warning: https://wyday.com/blog/
However, as you might've gleaned from my responses elsewhere in this topic, this was not a normal circumstance.
Here's a quick breakdown of what has happened so far:
1. We were investigating unusual communication to/from our servers.
2. It quickly became apparent to us that *at least* 3 large customers had embedded their web API key either in their app(s) or in client-side JS. And other customer had suspicious usage (meaning, they *likely* had used their web-API key in their app(s) or in client-side JS).
3. We saw no evidence of malicious 3rd parties using these leaked keys to steal data and/or manipulate data. However, knowing that it was *possible* steal data using the self-leaked web API keys introduced 2 things: legal responsibility & liability.
In other words, because we *knew* X web API keys had been self-leaked (i.e. leaked by programmers ignoring all documentation and warnings) we had to ensure that those keys could no longer be used. And because we suspected Y other web API keys had been self-leaked, we also had to block those keys.
4. Because legal liability and responsibility were introduced into the equation we now had a ticking time-clock. We had to fix these customer-imposed problems as quickly as possible with as minimal an impact as possible. And that's what we did.
The following use-cases were not impacted at all:
i. Our customers using the web API key on a single server. (It continued to work and will continue to work with no change required by our customers).
ii. End-users (your customers) using TurboActivate, TurboFloat, or the TurboFloat Server. Absolutely no interruption of service was introduced and none will be introduced.
However, due to time-constraints and the introduction of legal responsibility / liability to ensure user-data was safe and remained safe, the following use-cases *were* impacted:
a. Our customers using the web API key in their apps. The fact that we impacted these customers was the whole purpose and we have zero regrets about this.
b. Our customers using the web API key in internal-use apps for support staff, managers, etc. This was an unintentional consequence, but again (I've stated this above 3 times already), this type of usage violates our terms of service. And the solution is to create LimeLM accounts for these people.
c. Our customers using the web API keys in multiple back-end servers for ordering / querying / modifying licenses. This was also unintentional (but unavoidable due to time-constraints). Scott fell into this category, a handful of other customers fell into this category, and I'm guessing you fell into this category as well. I'm sorry you were effected by our decisions. These customers (you included) were unintentionally punished for the behavior of unrelated lazy programmers.
5. The fallout (this forum topic was created accusing us of incompetence, amateurish behavior, etc.).
6. [Happening before & during #5] We were doing further analysis see exactly which web API keys were effected and black-list them manually.
7. The partial-rollback: after manually blacklisting the known offenders we were able to "un-press" the "red button" to fix well-behaving customers (4c -- Scott, you, and a small handful of other customers) while keeping the known-offenders blocked.
That's the timeline thus far (as of this moment). So, what's coming next?
We've been working hard to improve our blocking of *potentially* compromised keys (i.e. keys that we cannot with 100% certainty say are being mis-used). This will happen on Monday. We've been working hard on a flexible solution. The current approach being tested will allow a single web API key to be used from 3 separate IPs within 72 hours. We have not finalized the design (we'll be working longer hours too to finish this).
More details will be on our blog after we make this change.
Very long story short: this was an atypical case handled as well as an atypical case can be handled. I am genuinely sorry some legitimate customers got effected.
As a quick follow-up for those generating keys from FastSpring, I also received a concrete confirmation that license generator scripts should only present as a single public IP address.
Wyatt wrote:> We've been working hard on a flexible solution. The current> approach being tested will allow a single web API key to be used from 3 separate IPs> within 72 hours. We have not finalized the design (we'll be working longer hours too> to finish this).
Wyatt, any update on a more flexible solution, both the technical details and ETA? FastSpring seems fine, but my Heroku-based license management features are having issues now, and Heroku doesn't have a first-class way to ensure a static IP address. There are third-party services I can use as effective proxies for outbound traffic, but before I make those types of changes I wanted to see if you guys were perhaps going to roll out something that would obviate the need for such a solution.
A more flexible version will be rolled out in the next 2 weeks (it requires some data-structure changes and extensive testing).
Currently: the web API can only be used from a single source IP address per 24 hours. This "timer" and last IP address can be reset manually on your settings page: https://wyday.com/limelm/settings/#api-key
More flexible solution (coming soon): you'll be able to specify how many IP addresses you want a web API key to be access from (0 to 3), and the time limit will be X times 24. (For example, if you set the limit for the web API key to be access from 3 separate IP addresses then the time limit will by 3*24 = 72 hours. Meaning the web API key can only be accessed by 3 separate IP addresses in that 72 hours). This more flexible solution will also be re-settable (like the current solution is).
In the meantime it looks like there are several solutions for Heroku regarding static IP addresses. For example: https://devcenter.heroku.com/articles/fixie
ScottIt was nice this 3-day break was over a weekend where we did not see this change until it broke today. Agreed with all of the others. Why were we not given a significant warning?
I use my API key in an app. The web interface really sucks in LimeLM. Things like the "extra data" field we want to search and we cannot. My own app allows me to do things like make changes and updates to multiple licenses, deactivate multiple licenses, etc. These are features that are just not available in the LimeLM web interface.
A co-worker also also does use my App. See, I can do things like have data verification in my app that cannot be done from the LimeLM web interface. The LimeLM interface is slooowwww, specially when are you creating a large number of licenses, as we do.
It is a concern and a security risk that I do have this API key in an app which could be discovered. Someone could get my key and do harm to my data. It would suck, but the benefits outweigh the risks. Yes, it said do not use the API in your apps, but we choose to because of the benefits out weigh the risks. You also provided C# samples which seems off to provide if you did not want us using the API in our own C# applications.
I use my app from multiple computers. I have a desktop which I do much of my work from. I have a laptop which also gets used a lot specially when I am in meetings. I also do work from home and have my app installed there.
Hey Scott,
>> "It was nice this 3-day break was over a weekend where we did not see this change until it broke today. "
We initially required 1 IP address per web API request on Monday (June 25th) evening. We did a partial rollback on June 27th in the morning leaving 3 full business days (5 days total if you include the weekend), plus part of today to implement fixes.
>> "Why were we not given a significant warning?"
Reason why it was rushed is here: https://wyday.com/forum/t/4205/current-limelm-server-status/#post-19954
Short answer: unique circumstances that put user's personal information at risk and required an immediate solution.
Shorter answer: critical security issue caused by customers who ignore documentation and warnings.
>> "The web interface really sucks in LimeLM. Things like the "extra data" field we want to search and we cannot."
Agreed. The new LimeLM interface will be out this summer.
>> "The LimeLM interface is slooowwww, specially when are you creating a large number of licenses, as we do."
Huh? It's as fast as a web API call. (The underlying tech is identical).
Can you explain exactly what you perceive as slow?
>> "It is a concern and a security risk that I do have this API key in an app which could be discovered. Someone could get my key and do harm to my data. It would suck, [...]"
Yes, that's an understatement.
>> "[...] but the benefits outweigh the risks."
No, they do not. If you do business in the E.U. (i.e. you're based in the E.U. or you have customers from the E.U.) you've introduced yourself to huge liabilities. See GDPR fines (google it).
>> "You also provided C# samples which seems off to provide if you did not want us using the API in our own C# applications. "
Because ASP.NET is a popular technology; not to use the web API from your app. We've always told people explicitly not to and told them why. We've also always had a warning on the settings page where you get your web API key.
>> "My own app allows me to do things like make changes and updates to multiple licenses, deactivate multiple licenses, etc. These are features that are just not available in the LimeLM web interface. "
Nope all those things are currently available in LimeLM and have been since day 1. Select multiple keys and click the "Edit" icon next to anyone of the selected keys. This will bring you to a page where you can mass revoke / unrevoke keys. Or mass-edit keys. Or mass-delete keys.
Need to only do that to specific keys? Use advanced search, then select the keys you want to edit, click the "Edit" icon next to anyone of the selected keys.
This will be more "discoverable" in the new LimeLM interface, but this is something we've always had.
Felix_Hi Wyatt, 1. We are hosting cloud based services which are automatically scaled. No thing like "one API key by IP address" as each request could theoretically be handled by another server instance. 3 will not be enough either.
2. You state: an API key can be used only by one person (yes, person), I can find 100 use cases where someone is using multiple IP addresses.
3. It broke our automated activation service from one moment to another without any information forehand. Thats very worrying for us...
>> "1. We are hosting cloud based services which are automatically scaled. No thing like "one API key by IP address" as each request could theoretically be handled by another server instance. 3 will not be enough either."
If you're talking about Heroku, there are several options (already mentioned in this thread). Simply googling "[Your Host Name] static IP address" will give you more information about setting things up correctly.
>> "2. You state: an API key can be used only by one person (yes, person), I can find 100 use cases where someone is using multiple IP addresses."
IP addresses are a weak indicator of anything. We know this. This is why they are completely ignored in the activation process: https://wyday.com/limelm/features/why/#wrong-id
However, web API keys are not activated. They're basically a username / password all-in-one.
Long story short: it's the best solution we have right now to prevent our customers from ignoring our documentation and leaking all of their customers' information.
Creating a server with a static IP address and using the web API key from that server is the typical use-case. For everything else, log into your LimeLM account using your username and password.
>> "3. It broke our automated activation service from one moment to another without any information forehand. Thats very worrying for us..."
I'm sorry that happened. We tried to notify everything we thought would be effected, however we missed a number of customers (our sample size for the detection was *way* too small).
The reason for the rollout is described above (multiple times). Long story short: security improvements required by law to ensure end-user data is safe and remains safe.
Online activations and verified trials were unaffected (and will remain unaffected). If you're talking about an automated offline activation form, yes, if the servers you had that hosted on had dynamic IP addresses then those request would be blocked. However the customer could still online activate or manually send your company offline activation request.
AndrewSWyatt, we use microsoft Azure which does not guarantee which IP address it will use for outgoing requests. It does stick to a range of address however 3 is not enough for this as it uses a range of 1 out of any 5. Can I request that your new changes allow 5 ip addresses instead of 3. Alternatively we know the specific IP addresses it will use. So maybe allowing us to specify the authorised IP numbers that can use a particular API is the solution?
This is a major problem to our business as we generate and issue all trial license keys from our Azure hosted website. We are having major complaints that users cannot obtain a key for our software and we are loosing business as it is quickly put into the too hard bucket for our clients.
I would say we are currently having to manually reset the ip number listing about every 15 mins to keep some kind of service to our clients going. It is not a long term solution.
I have created a user for our Fastspring service and I am now using a different API key for that as suggested and it seems to be working but I cannot do anything about the 1 in 5 chance on the Azure platform.
When do you think your new solution will be available?
Hey Andrew,
The solution (as stated above) is to use a static IP address with whatever hosting provider you're using. Most provide static IPs out of the box. Some require enabling the feature. Googling "[hosting provider] static IP address" will get you the directions for your particular hosting provider.
For example, this query brought back plenty of options for setting a static IP:
https://www.google.com/search?q=azure+static+IP+address&ie=utf-8&oe=utf-8
https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-deploy-static-pip-arm-portal
https://azure.microsoft.com/en-us/pricing/details/ip-addresses/
Our more flexible solution for API requests (max 3 for 72 hours) is not available yet, but I can tell you that even when we roll that out you'll need to use a static IP address.
ScottMy company uses Netsuite for our sales management. Using Netsuite, we have automated our license creation process. This saves a tremendous amount of work and reduces chance of human errors. Since Netsuite uses multiple IP addresses and we cannot control these, we are being reduced to once again manually creating licenses, which is very time consuming.
Ignoring all of what I wrote below, the crucial piece is our sales system which is being blocked now. This is very concerning.
> >> "It was nice this 3-day break was over a weekend where we did not see> this change until it broke today. "
> We initially required 1 IP address per web API request on Monday (June 25th) evening.> We did a partial rollback on June 27th in the morning leaving 3 full business days (5> days total if you include the weekend), plus part of today to implement fixes.
> >> "Why were we not given a significant warning?"
We were luck enough not see the issue on June 26. You disabled things on June 27, but come Monday, you enabled this check again and we are suffering from this.
> Reason why it was rushed is here:> https://wyday.com/forum/t/4205/current-limelm-server-status/#post-19954> > Short answer: unique circumstances that put user's personal information at risk and> required an immediate solution.> > Shorter answer: critical security issue caused by customers who ignore documentation> and warnings.
> >> "The web interface really sucks in LimeLM. Things like the "extra> data" field we want to search and we cannot."> > Agreed. The new LimeLM interface will be out this summer.
It is not out now. We have been relying on my app for the last couple years because of this.
> >> "The LimeLM interface is slooowwww, specially when are you creating a> large number of licenses, as we do."> > Huh? It's as fast as a web API call. (The underlying tech is identical).> > Can you explain exactly what you perceive as slow?
Yes, the calls are quick. What is slow is making each call. For example, I want to update 100 custom fields in different license keys. My own tool, I can do this in about 5 seconds time. Using LimeLM web interface, I have to go through multiple steps to select the search query selecting multiple drop downs, which is time consuming. I am limited to ONLY performing the queries you allow us and there is no data verification. We then have to select "wildcard" and enter asterisks. My app, avoids all of this and I can do AND or OR searches. My search results, are copy friendly allowing me to easily grab the license key or any of the custom fields associated with the license. Web interface, nope. This is an example of one small search. If you saw my app you would understand. Simply, there is things I can do that your web interface simply does not come close to.
On top of that, I can query the extra data. Your web interface, you cannot and this is crucial to us.
This is not an app that is using TurboActivate that we are distributing to customers. This is something that is used a couple people so we can manage our licenses. We have safeguards in place to minimize the chance of someone getting this tool who should not. We use strong passwords, change them often, use drive encryption, etc.
> > >> "It is a concern and a security risk that I do have this API key in an> app which could be discovered. Someone could get my key and do harm to my data. It> would suck, [...]"> > Yes, that's an understatement.> > > >> "[...] but the benefits outweigh the risks."> > No, they do not. If you do business in the E.U. (i.e. you're based in the E.U. or you> have customers from the E.U.) you've introduced yourself to huge liabilities. See> GDPR fines (google it).
In your opinion, it does not. In ours, it is.
> > >> "You also provided C# samples which seems off to provide if you did not> want us using the API in our own C# applications. "> > Because ASP.NET is a popular technology; not to use the web API from your app. We've> always told people explicitly not to and told them why. We've also always had a> warning on the settings page where you get your web API key.> > > > >> "My own app allows me to do things like make changes and updates to> multiple licenses, deactivate multiple licenses, etc. These are features that are> just not available in the LimeLM web interface. "> > Nope all those things are currently available in LimeLM and have been since day 1.> Select multiple keys and click the "Edit" icon next to anyone of the> selected keys. This will bring you to a page where you can mass revoke / unrevoke> keys. Or mass-edit keys. Or mass-delete keys.> > Need to only do that to specific keys? Use advanced search, then select the keys you> want to edit, click the "Edit" icon next to anyone of the selected keys.
> This will be more "discoverable" in the new LimeLM interface, but this is> something we've always had.
I understand you can do some of this on the web interface. You do not understand how difficult it is to do this. Each time you select deactivate you move to another web page and have to scroll through the list again. I know you can select the keys you want to mass edit but if you are not careful and click outside the check box, everything you checked becomes unchecked. It is painfully slow.
Again, we have spoken about this in the past. I would be more than happy to show you my experiences searching directly in LimeLM.
>> My company uses Netsuite for our sales management. Using Netsuite, we have automated our license creation process. This saves a tremendous amount of work and reduces chance of human errors. Since Netsuite uses multiple IP addresses and we cannot control these, we are being reduced to once again manually creating licenses, which is very time consuming.
Shoot me an email at wyatt@wyday.com. Tell me how many IP Addresses, whether youve changed the web API key (to prevent the app you release to your customers from using the web API).
Long story short: using the web API in your backend processes is supported, and we want to make the right balance of security an ease.
Using the web API key in an app that reaches your end-users was never supported, however we made the mistake of assuming our customers would actually read and follow our instructions. That was naive. Now we enforce it so customers dont shoot themselves in their feet.
YepWyatt, I don't think your derisive language towards your customer base is particularly helpful. You've now got 7 pages of customers using your product in legitimate ways who have been broken without warning, and in those 7 pages I don't see any who were doing the wrong thing (distributing the API key embedded in their apps). Yet you still trot out lines like "we made the mistake of assuming our customers would actually read" as if that has any bearing on the genuine issues being raised.
YepI do appreciate that you have listened to us and are working on a change that will alleviate some of our concerns (allowing 3 IP addresses for a 3x24hour window). LimeLM is a great product and I've recommended it to a fair few peers and acquaintances over the years. I hope I can continue doing so. Thanks.
AndrewSHi Wyatt, We have one app which uses a unique API key. We do not pass round different users however it not a simple as you suggest with Azure. You can google all day long and you will cerntainly get instruction on how to set up an Azure static ip address, but ONLY inbound! Just for information we do have a static IP number (inbound, as Azure allows)
Apologies for the capitals but it is the only way to highlight text on this forum.
He is an extract from the Azure help article about out bound IP addresses.
Directly from Azure:
<When outbound IPs change
Regardless of the number of scaled-out instances, each app has a set number of outbound IP addresses at any given time. Any outbound connection from the App Service app, such as to a back-end database, uses one of the outbound IP addresses as the origin IP address.
*****You can't know beforehand which IP address a given app instance will use to make the outbound connection, so your back-end service must open its firewall to all the outbound IP addresses of your app.*****
The set of outbound IP addresses for your app changes when you scale your app between the lower tiers (Basic, Standard, and Premium) and the Premium V2 tier.You can find the set of all possible outbound IP addresses your app can use, regardless of pricing tiers, by looking for the possibleOutboundIPAddresses property. See Find outbound IPs.>
As you can see from above Azure DOES NOT allow you to specify a static ip number for outbound connections. I will state again it does tell you the 5 IP number it may use which was why I requested you make the new limit 5 address or give us the ability to specify the exact addresses.
Azure users do seem to be a significant and growing client base you are alianating here from API access to LimeLM.
In the best interests of your business and mine I request once more you re-think your workaround solution in a bit more detail with this possibly new information about Azure.
>> "I don't think your derisive language towards your customer base is particularly helpful. You've now got 7 pages of customers using your product in legitimate ways who have been broken without warning, and in those 7 pages I don't see any who were doing the wrong thing (distributing the API key embedded in their apps). Yet you still trot out lines like "we made the mistake of assuming our customers would actually read" as if that has any bearing on the genuine issues being raised."
The problem (a handful of customers self-leaking their web API key and putting their data and their customers' data at risk) was created by customers who were either:
a. not reading the explicit warning to treat the web API key like a password and not embed it into their app.
or
b. they read the warning, understood it, ignored it, and put their customers' data at risk.
Those are just the facts. I wouldn't say that's derisive. It's blunt, no doubt, but I also go out of the way to fault ourselves saying we were naive to assume customers wouldn't choose the easiest path presented, even if it negatively effected their customers' data.
As I've said above about 6 times already, this irresponsible use of the web API keys (using it in a customer-facing app) forced our hand (i.e. the hard limits, rolled out quickly, etc. -- see above). Why? Because as much as it's *your* job to protect your customers' data, it's also *our* job to protect your customers' data.
Not only is it our legal responsibility (re: GDPR and privacy regulations rolling out in other countries) it's our moral responsibility.
>> "As you can see from above Azure DOES NOT allow you to specify a static ip number for outbound connections. "
It does, unfortunately their documentation is often contradictory or doesn't link to updated help articles:
https://stackoverflow.com/a/34320979/124805
https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-reserved-public-ip
No ETA. You should use a separate web API key for every server you use the web API key on. And you should use static IP addresses for the servers for which you use the web API key on.
The flexible solution we're in the process of creating is *only* for cases where you don't have full control over the infrastructure (i.e. a case like using the web API key from FastSpring). It's not so you can use the same web API key on separate servers.
ColinWe use Salesforce CRM in our back office to generate/amend keys via LimeLM web api. Like Azure users (and others) we have no control over the IP address Salesforce's servers use. Fortunately it is fairly static but it can change. We were caught out today by an IP address change. The proposed change to allow a limited number of changes per 24 hour period would certainly fix it for us.
JasonThis broke my license payment system and now I have to go through the logs, find all the people who paid for a license and didn't get one and fix their situations.
Sending us an email would have prevented this.
YepWyatt wrote:> No ETA. You should use a separate web API key for every server you use the> web API key on. And you should use static IP addresses for the servers for> which you use the web API key on.
And if this isn't server-side software? I have a desktop admin app and CLI that want to use on my office PC, my office laptop, my home PC, and my home laptop.
Log into your account and just use it. The LimeLM interface can do everything the web API can do, and do it more efficiently.
Or, if the muscle memory of your custom apps is too strong, log in to your LimeLM account and reset the last used IP address. Just a word of warning, were going to limit the number of resets per day (people are *already* abusing this functionality)
YepThe admin app tie together a number of different systems. LimeLM is just one of those. So no, just using the web UI isn't an acceptable answer. The API was an advertised feature of LimeLM, and part of the reason we went with your product. What are our other options?
I gave you 2 options. Read the second paragraph for the option where you still use your app from multiple different machines.
YepThanks Wyatt, but that's not an acceptable solution, either. We'll just wait for the fix you spoke about earlier.
JasonWyatt,
I don't know how many users you have, but I still haven't gotten an email about this change, so it sounds like it's possible there are still users out there who are affected by this change and aren't aware of it. They could be losing customers and money because of this.
An email from you to them would prevent this. Is this really too much to expect?
Should we have scripts to test the API every day? Or are we expected to check this forum every day?
I don't object to changing the API for security requirements, even with short notice. But you STILL have not notified users.
Jason
Hey Jason,
I saw your email but haven't had a chance to respond. Since you're asking the same questions here I'll answer you here.
>> "An email from you to them would prevent this. Is this really too much to expect?"
As stated above, this was an atypical change that needed to be released quickly. Other changes are not rolled out like this.
We tried to gather an accurate view of customers that might be effected and we missed a few accounts (because our time frame was too short to handle low request amounts).
Going forward you should add error handling to your backend. So, if request to LimeLM fail for whatever reason you should notify yourself and your team. Our examples say as much, but we don't show how to do it because it depend upon the internals of your organization.
>> "Should we have scripts to test the API every day?"
Just add error handling. You should know when things fail for whatever reason. Just "swallowing" errors and assuming things are running smoothly is never recommended.
>> "But you STILL have not notified users. "
We have. As stated above we notified the users that looks like legitimate use and we thought might be effected. Our problem was sample-size dictated by the fast-rollout.
We don't spam all users via email about changes. We never will. It's a waste of our time and most users' time.
Any typical "breaking" changes to the web API or otherwise will be announced on the blog. https://wyday.com/blog/
(This was an atypical case -- hence the quick action -- see above for a detailed description).
Use your favorite RSS reader, or check in daily. Whatever is more time efficient.
When the new LimeLM interface is released we'll also have notifications in the interface. Again, this presupposes you login to your LimeLM account. Some users have written alternative LimeLM interfaces using the web API -- and those users will miss these change notifications.
Jason"As stated above, this was an atypical change that needed to be released quickly. Other changes are not rolled out like this."
I understand you had to roll it out quickly with no notice. What I object to is that you didn't give notice AFTER, even weeks later, even though it was clear this was affecting people negatively.
"We tried to gather an accurate view of customers that might be effected and we missed a few accounts (because our time frame was too short to handle low request amounts)."
Just email your users. I don't understand what is so hard about this.
>> "But you STILL have not notified users. "
"We have. As stated above we notified the users that looks like legitimate use and we thought might be effected. Our problem was sample-size dictated by the fast-rollout.We don't spam all users via email about changes. We never will. It's a waste of our time and most users' time."
I have not received an email from LimeLM, other than a reply to one I sent or a "we've delivered your purchase" email, since 2011. I am not worried about spam from LimeLM. I'm worried about not being aware of changes.
I'm sorry to belabor this but it just seems so simple to email us when something changes - even after the fact if a change is that urgent.
datalockHi Wyatt,
Most other API system have custom permissions of the API key. Is this something you can put in place and still keep the security aspect of new changes? For instance, 1 API key can only perform less damaging actions such as update custom fields and not create or delete pkeys.
We use a web app system that scales using dockers and each of the IP Address are never the same. We have relied on your license system for years and have implemented your system in many of our products. Not having the ability to automate with the web API will cause us to look for alternatives.
Also we did not get notified of the change. Of course we were only notified by our disgruntle customers and we lost half the day locating the problem. Would have been nice to give us a heads up so we can notify our support team of the potential issues.
>> "For instance, 1 API key can only perform less damaging actions such as update custom fields and not create or delete pkeys."
We already have permissions like that available: https://wyday.com/limelm/help/add-users/
That wasn't the problem (nor the solution to the problem). The actual problem is described above multiple times, but briefly, some of our customers (both large and small companies) were putting *their* customers' data at risk. It's their job *and* our job to protect their data and their customers' data.
We had (and continue to have) both a moral and legal obligation to ensure the safety of our customers' data and our customers' customers' (i.e. the end-users') data.
Hence the change and the quick rollout.
>> " Not having the ability to automate with the web API will cause us to look for alternatives."
The ability to automate using the web API is still possible. It just requires you use either a typical server with a static IP address (available cheaply from multiple providers -- this isn't a huge cost). Or if you want to continue to spin up docker containers or other equivalent VMs / dynamic solutions, then use static IP addresses for the docker / VM / dynamic solution.
Protecting your data and your customer's data is a legal requirement. Any other licensing solution either already tightly restrict web API usage or will do so in the near future.
Also, as stated above multiple times, TurboActivate, TurboFloat, and TurboFloat Sever are completely unaffected by this change.
>> "Also we did not get notified of the change."
As stated above, we did a sample of web API requests over a period of time (that turned out to be too small) and notified those customers who we believed to be effected. I'm sorry our small sample size excluded you from the notification. In the future we'll make all potentially breaking change announcement available in LimeLM interface and in a blog post: https://wyday.com/blog/
On your end, as stated above multiple times (and recommended in our documentation and examples that use the web API), you should enable reporting so that you're notified of web API failures. I.e. don't "swallow" errors or exceptions.
YepHi Wyatt,
Any idea on the time frame for the release of the change that will allow us to use 3 IP addresses on a single API key?
Thanks
Hey Jimmy,
No hard date. This flexible solution (max 3 IPs per 72 hours) is not a solution for hosts with dynamic outgoing IP addresses. Its only a solution for hosts with static IP address that you dont have full control over (e.g. FastSpring).
YepThanks Wyatt,
That's fine; we have a service that can run from exactly 3 different IP addresses, so the number will work just fine for us. Our long term plan is to change things to use different API keys as you have suggested, but we don't have the resources to do that any time soon (it would have been nice to get some notice!). I appreciate that you can't give a hard date, but can you let us know if we are talking days/weeks/months here?
Thanks,Jimmy
Likely weeks.
----
Edit (2019): For a full breakdown of this "issue" and our choices, see this blog post: https://wyday.com/blog/2019/when-in-conflict-security-supersedes-usability/
Further comments can be made on that blog post. Closing this topic.
