How to Debug Node.js HTTP Requests

Yesterday I got quite frustrated while programming. Attempting to start an implementation of a Slack Application, I ran into an issue that has plagued me since starting web development: The black box that is other people’s API’s and request packages like Fetch and Axios.

More specifically, I spent hours scouring forums, decoding error objects and tearing my hair out, only to find that my POST request was sending no data at all.

In this article, I want to explain what I have learned regarding this specific issue, as well as several great ways I have found to generally debug your API requests. Moreover, I hope by codifying these instructions, I, personally, will never make these mistakes again.

Know your error

The first step in figuring out why your API call isn't working is knowing what your error means. This is gruelling, but you have to read the documentation. The whole thing.

Small text can hold information to solve your entire bug, so if you just skim everything, you could be wasting time, not saving it. This, and everything else in this article, is exemplified with this OAuth method from Slack’s Application API.

Always read the fine print

This message held one of the keys to my bug, but if I had just ⌘F’ed my error message, I would have been misled.

Documentation directly contradicting itself

The moral of the story is to read the documentation … I know, controversial. Note this issue with Content-Type will come up later.

Don’t trust Web Developers

This next point might be a bit rich coming from myself, a web developer, but I know as well as anyone that no one really knows how to check for errors. The following two error messages are subtly different, can you see it?

Error object 1

Error object 2 (exactly the same)

That’s right! They’re the same error from two completely different requests with different problems. The first message is from an access code that is beyond its lifespan, so therefore an invalid_code. The second one, however, is from a POST request with no data at all. This is caused by someone checking if “the access code is invalid or doesn’t exist” which is lazy, and in so doing sent me off on a wild goose chase trying to find formatting I was supposed to be doing to my access code.

This had me scratching my head for hours. “How is my code invalid, I put it directly from the querystring into my request, it's not my fault.” Everyone knows when you’re writing the code, it’s always your fault. So let’s talk about how to find an error when you cannot see the process of throwing it.

Step One in this situation is to ignore the specific error code. Computer Scientists are lazy, if something doesn’t make sense, you can throw it out and do your own debugging.

Step Two, and this should always be your step two, is to put it into Postman.

Postman

Postman is amazing. In the next life, I want to be reincarnated Post-Man. For those who don’t know Postman is a service to send out HTTP requests. Simple as that.

The following is an example of a Postman POST request. Surely the name is a coincidence.

An example POST request with body

A successful POST response

So we have now made a POST request manually, and it was successful, telling us all of our data, including our supposedly invalid_code, is correct. While this is a step in the right direction, this is possibly the most infuriating part. I repeatedly ask myself while making calls to any API: Why is this working in Postman but not in my code? Usually with a few expletives thrown in.

Postman Echo

Our Third Step is a tool that I found in this round of debugging. It is a tool called Postman Echo, a service that does what it says on the tin. It Echos back your HTTP request.

var newdata = {
code: "************",
client_id: "************",
client_secret: "************",
redirect_uri: "************"
}
fetch(`https://postman-echo.com/post`, {
    method: 'POST',
    body: JSON.stringify(newdata),
    headers: {
        'Content-Type': 'application/json; charset=utf-8',
        'Access-Control-Allow-Origin': '*',
    }
})
    .then(res => res.json())
    .then(json => console.log(json))

You should simply delete your API’s URL from your request and replace it with the Echo API’s link. This solves the aforementioned black box issue with request libraries such as Axios and Fetch, by giving you insight into exactly what the server is being given.

If this POST / GET stuff goes over your head, it’s a bit technical but while learning this stuff I viewed this Mozilla article almost every session 😉.

Postman Echo didn’t return my data

Here we finally arrive at our issue. Axios and Fetch make passing in malformatted data entirely too easy. No error message, no warning, no idea what is happening. I found myself repeatedly googling “axios body always empty” or “Fetch post example with parameters” to no avail. That is until I got a message from above (my dad) telling me to look within (the request) and speak true (format my data correctly.)

An important facet to note is the Content-Type header. Usually API’s will require Content-Type: "application/x-www-form-urlencoded" or Content-Type: "application/json". Feel free to copy and paste either of those, not both, into your headers object.

What neither Axios nor Fetch warns you about at runtime, is that both of these require manual user formatting which is the absolute root cause of our problem.

Fetch

For application/json node-fetch requires you to format your data with a JSON.stringify(data). And for application/x-www-form-urlencoded it wants you to use qs.stringify(data). The generally accepted package is qs.

var data = {
    code: "************",
    client_id: "************",
    client_secret: "************",
    redirect_uri: "************"
}
fetch(`https://postman-echo.com/post`, {
    method: 'POST',
    body: JSON.stringify(data), // change to qs.stringify(data)
    headers: {
        'Content-Type': 'application/json; charset=utf-8',
        "Access-Control-Allow-Origin": "*",
    }
}).then(res => res.json())
.then(json => {
    console.log(json) 
    // your data is in json.data or json.form (see below)
})

Something else to note is if you use application/json your data will exist in json.data, whereas with application/x-www-form-urlencoded it will exist in json.form.

Axios

As for Axios, no manual formatting is required, but I found in writing this article, that res.data will not always give you POST data. As with Fetch, application/x-www-form-urlencoded is in res.form. All the same, here is an example of an Axios POST query with headers and parameters.

var data = {
    code: "************",
    client_id: "************",
    client_secret: "************",
    redirect_uri: "************"
}
axios.post(`https://postman-echo.com/post`, data, {
    headers: {
        'Content-Type': 'application/json; charset=utf-8',
        "Access-Control-Allow-Origin": "*",
    }
}).then((res) => {
    console.log(res.data.data) // for urlencoded use res.data.form
})

If you are using your own API

If you have reached the end of these sections (thank you for reading) and you still have your bug in full swing, then your problem might be with Express. I won’t go too much into this, but make sure you are using middleware to allow for parsing request bodies. See this StackOverflow article for a bit more info.

Otherwise, for the long and short of it, put these two at the top of your code:

app.use(express.json())
app.use(express.urlencoded())

They are not mutually exclusive so using both can’t hurt. It is quite stupid that these aren’t automatically set up in Express, but hey ho.

What a journey we have been through

If you have gotten this far and haven't just stolen my code and left, thank you for reading. I hope this process will save some hair follicles for someone out there.