API POST request format?Answered

It really depends on the programming language you’re using to make the request. What programming language are you using to make the web API requests?

Also, what library in the programming language are you using? Usually there are multiple options per language.

Thank for the reply, Wyatt. I'm using Java/JAX-RS/Jersey, but given that these are just HTTP requests, the answer to the question I'm asking should be completely independent of the programming language/library. I'm not asking how to issue a POST request in the client implementation language, but rather how should the POST request parameters be included vs. how they're included for the current GET requests, specifically which ones should be included as query parameters vs. headers vs. in the POST body? And if sent as headers or in the POST body, what is the desired/expected format of those parameters?

I've already verified that I can successfully switch my current GET requests to use POST with everything as query parameters and an empty POST body and things work, but that means that sensitive data like the API key is still included in the query string which is of course not encrypted when requesting over HTTPS/SSL. That seems contrary to the security goals of requiring all requests via POST/HTTPS going forward. I would think that the query string for a POST request would include at most the method and format parameters, and everything else would be sent as something that is encrypted in-transit.

Let's use limelm.pkey.getID as an example. For a GET request it looks like:

GET https://wyday.com/limelm/api/rest?api_key=apiKey&format=json&method=limelm.pkey.getID&pkey=licenseKey

What should the corresponding POST request look like? As I mentioned, I've verified that using the exact same URL but a method of POST and an empty POST body does work, but I would think it would support at least the sensitive query params being sent as headers or in the POST body, e.g.:

POST https://wyday.com/limelm/api/rest?format=json&method=limelm.pkey.getID

Headers or POST body
api_key=apiKey
pkey=licenseKey

Or should the POST body be formatted as XML/JSON? If so, what is the expected/required format?

Please let me know if that doesn't clarify the question I'm asking.

Everything should be POSTed. Only the URL (https://wyday.com/limelm/api/rest/) should be in GET part of the request.

The problem this is solving is overly ambitious middleware-loggers. Namely, prevent credential & privacy leaking when corporations do stupid things like log encrypted requests — which happens, hence this change.

So, everything should be in the POST body. And it can be posted as an URL formatted string (so, take everything you were using as GET parameters and use them as POST parameters). It should be a 1 or 2 line change depending on the library you’re using.

Use this function to test the new stricter behavior: https://wyday.com/limelm/help/api/limelm.test.echo/

All GET parameters are ignored.

Does that make sense?

What you've said makes sense and matches my expectations for such a security-driven change, yes, but it doesn't match my findings:

All GET parameters are ignored.

That doesn't seem to be the case right now. I changed my current client to use POST instead of GET but with no other changes--specifically it still uses the query string to pass all parameters and doesn't provide any POST body--and it works just fine. It doesn't seem like query parameters (i.e., “GET parameters”) are ignored right now when the method is POST. I'll provide a concrete example below.

So, everything should be in the POST body. And it can be posted as an URL formatted string (so, take everything you were using as GET parameters and use them as POST parameters).

That was actually the first thing I tried. Again, to be perfectly clear, I changed my client to issue a POST request against:

https://wyday.com/limelm/api/rest

with the following as the POST body:

api_key=apiKey&format=json&method=limelm.pkey.getID&pkey=licenseKey

 and it failed with:

<?xml version="1.0" encoding="utf-8"?>
<rsp stat="fail"><err code="101" msg="Method not provided."/></rsp>

I've tried the same thing against limelm.test.echo and that yields the same error response.

To be even more concrete, here's pretty much exactly how I've been issuing the request via GET which works just fine:

WebTarget client = ClientBuilder
    .newClient()
    .target("https://wyday.com/limelm/api/rest")
    .queryParam("method", "limelm.pkey.getID")
    .queryParam("format", "json")
    .queryParam("api_key", apiKey)
    .queryParam("pkey", licenseKey);
Response response = client.request().get();
// Process the JSON response

and as stated, it also works fine when using POST with query params on the URL and no POST body, i.e.:

WebTarget client = ClientBuilder
    .newClient()
    .target("https://wyday.com/limelm/api/rest")
    .queryParam("method", "limelm.pkey.getID")
    .queryParam("format", "json")
    .queryParam("api_key", apiKey)
    .queryParam("pkey", licenseKey);
Response response = client.request().post(null);
// Process the JSON response

but trying to issue a POST with the query string used as the POST body yields a “Method not provided” error:

WebTarget client = ClientBuilder
    .newClient()
    .target("https://wyday.com/limelm/api/rest");
Entity<Object> postBody = Entity.text(
    "method=limelm.pkey.getID" +
    "&format=json" +
    "&api_key=" + apiKey +
    "&pkey=" + licenseKey
);
Response response = client.request().post(postBody);
// Fails with "101: Method not provided"

Based on what you've said, though, that's how it should work, correct? Apologies if I'm still missing something simple here.

And note that I've reproduced this exact behavior using other HTTP clients such as curl and the integrated JetBrains HTTP client, e.g. the following request in the JetBrains HTTP client (where apiKey and licenseKey are properly specified, of course):

POST https://wyday.com/limelm/api/rest

method=limelm.pkey.getID&format=json&api_key=apiKey&pkey=licenseKey

yields:

HTTP/1.1 200 OK
Date: Sat, 18 Dec 2021 21:57:53 GMT
Content-Type: text/xml;charset=UTF-8
...

<?xml version="1.0" encoding="utf-8"?>
<rsp stat="fail">
   <err code="101" msg="Method not provided."/>
</rsp>

and the following request:

POST https://wyday.com/limelm/api/rest?method=limelm.pkey.getID&format=json&api_key=apiKey&pkey=licenseKey

yields:

HTTP/1.1 200 OK
Date: Sat, 18 Dec 2021 22:20:40 GMT
Content-Type: application/json
...

jsonLimeLMApi({
 "pkey": {
   "id": <redacted>
 },
 "stat": "ok"
})

Let me know if that doesn't help.

Wyatt, given that you're going to begin alerting accounts via email about GET/HTTP API requests starting tomorrow, do you have any thoughts on what I've described here? Am I just not using the API correctly (and if not, can you tell me what I'm doing wrong?), or is the API itself not working correctly for POST requests right now (and if so, does that shift your posted timeline)?

I'm trying to get things squared away according to what you've requested, but as far as I can tell the closest I can get right now is to change the HTTP method from GET to POST and leave the params in the query string. I guess that might avoid getting email alerts, but it's certainly not the desired end state.

Wyatt, thanks for the pointer to the PHP example (and assuming that it came through briefly before I removed it, please ignore my false start on a Monday morning first response). Unfortunately I'm not really seeing anything in the PHP example that shows me what I'm doing wrong here. These two examples that I've provided:

Java

WebTarget client = ClientBuilder
   .newClient()
   .target("https://wyday.com/limelm/api/rest");
Entity<Object> postBody = Entity.text(
   "method=limelm.pkey.getID" +
   "&format=json" +
   "&api_key=" + apiKey +
   "&pkey=" + licenseKey
);
Response response = client.request().post(postBody);
// Fails with "101: Method not provided"

JetBrains HTTP client

POST https://wyday.com/limelm/api/rest

method=limelm.pkey.getID&format=json&api_key=apiKey&pkey=licenseKey

HTTP/1.1 200 OK
Date: Sat, 18 Dec 2021 21:57:53 GMT
Content-Type: text/xml;charset=UTF-8
...

<?xml version="1.0" encoding="utf-8"?>
<rsp stat="fail">
  <err code="101" msg="Method not provided."/>
</rsp>

should both be the equivalent of the following in PHP:

$request = curl_init('https://wyday.com/limelm/api/rest');
curl_setopt($request, CURLOPT_POSTFIELDS, 'method=limelm.pkey.getID&format=json&api_key=apiKey&pkey=licenseKey');
$post_response = curl_exec($request);
curl_close($request);

I've even tried adding nojsoncallback=1 as in the PHP example, but that doesn't make a difference. Note that I have other vendors' APIs that I invoke right now using POST in this same manner that all work just fine. Any insights you can provide into what's missing from the examples provided above--a required header perhaps?--are greatly appreciated because I fail to see why what I'm doing doesn't match what you're saying should happen or what is shown in the PHP example.

Well, the biggest hint I have is the “method not provided error”. This means that the POST parameters weren’t sent in a sane way to our servers.

So, whichever HTTPS posting library you use (honestly, use a thin wrapper over Curl if you can) you need to ensure that the data is sent as url-encoded (both the data must be properly formatted and not “magically cleaned” by whatever backend the library uses) and the url-encoded request header must be used.

Okay, I'm trying to narrow this down using just curl and have found something very curious.

Run locally from Windows, Mac, or Linux, this command:

curl -L -X POST https://wyday.com/limelm/api/rest -H "Content-Type: application/x-www-form-urlencoded" -d "method=limelm.pkey.getID"

yields:

<?xml version="1.0" encoding="utf-8"?>
<rsp stat="fail"><err code="101" msg="Method not provided."/></rsp>

Yet if I run that exact same command from https://reqbin.com/req/c-sma2qrvp/curl-post-form-example, it yields the expected:

<?xml version="1.0" encoding="utf-8"?>
<rsp stat="fail"><err code="100" msg="Invalid API Key (Key not found)"/></rsp>

The local network is very simple. There is no proxy, etc., involved. Obviously I use POST all the time from this network and don't see request parameters transferred via POST body removed or altered. What's unique about your APIs that might be resulting in this behavior? Do you happen to see these requests in your Web server logs? If so, do you see anything curious?

Sorry, but I'm just not seeing how to make progress on this given that I've never seen such an issue with any other service, but obviously I do see a difference in the same request issued locally vs. issued elsewhere. Any insights you can provide based on what you can see from the server-side of these requests are greatly appreciated.

UPDATE: I just tried running that same curl command from an external resource, specifically from an AWS VM on the same network where these API calls would be issued for real in production, and it yields the same results as when I run it locally. So it's not something specific to my network, and it seems that it would happen in my production environment as well. That doesn't explain why it looks better when run from reqbin, but it does show a consistent unexpected behavior even when using curl as you recommended.

Wyatt, I apologize but I'm still needing help here, and with a clock ticking, I'd strongly prefer to get resolution on this sooner rather than later to avoid any (what should be unnecessary) interruption in my service. As you can hopefully see from the previous two posts, even using curl I'm seeing inconsistent response behavior for the same command. I'm able to reproduce the “Method not provided” message with a curl command that clearly resolves to a method in some other instances. In my experience this type of inconsistent behavior is either due to something on the client/client network or is due to something in the way that the server processes these requests. I've now reproduced the errant behavior ("Method not provided") from a myriad of clients including:

• curl executed locally from Windows, Mac, and Linux
• curl executed on several completely separate networks from Linux which should rule out it just being my local network
• Postman
• Java-based client
• JetBrains HTTP client

Generally the next piece of diagnostic information that would be invaluable would be someone on your side looking at the Web server access logs--or even better, the actual request handler logs--to see why these requests don't seem to be processed properly except when run via curl from reqbin. Any help you can provide in support of your product and the upcoming API access changes would be greatly appreciated.

Yep, that was it. Thank you for finding that. Note that I've been using the former URL for GET requests all along, so that's why it never occurred to me to add a trailing slash to the URL that was already being used. It's very odd that a curl request for the former URL works from reqbin but seemingly not from anywhere else. Anyway, I'll get an update to my client pushed out once I've tested these changes.