Today, we will discuss ChatGPT and the dangers of blindly trusting code generated by artificial intelligence. The same risks exist when developers intentionally copy other developers' code without analyzing the code's actual functionality. How does ChatGPT's code quality fare when it comes to security?

Overview

If you are tech-savvy, you are probably already familiar with ChatGPT. However, if you are not, here is a brief definition of this fantastic tool created by OpenAI, taken from their website:

"ChatGPT is a sibling model to [InstructGPT], which is trained to follow an instruction in a prompt and provide a detailed response."

Due to the enormous volume of data and inputs that this AI has already received, it is able to answer a wide variety of questions, ranging from simple questions such as "How do I make a cake?" to more complex ones such as "How do I develop an endpoint in Node.js that receives a user, validates the name property, and saves it in the database?"

Because of its enormous power to provide answers, many programmers have started using ChatGPT as a research source for their day-to-day work. What previously required searching for four or five questions on StackOverflow is now easily answered by ChatGPT. This is much more practical, isn't it? However, not everything is perfect. As an AI, ChatGPT can and will eventually generate wrong answers. Therefore, in this document, we will examine some cases of wrong answers generated by ChatGPT and explain why developers should be very careful when copying code from ChatGPT or any other source.t.

Case 1: XSS Attack

In this document, we will examine an example of code generated by ChatGPT that has the XSS vulnerability. Specifically, XSS is a vulnerability that enables an attacker to run malicious code in your browser without your knowledge or authorization. In this case, I instructed ChatGPT to:

Create an endpoint that receives a user ID as a parameter in the URL. Look for the user in the database and return the request. If the user is not found in the database, indicate that the user was not found using TypeScript.

ChatGPT generated a working code that performs the requested function. The following is the code generated by ChatGPT.

At first glance, the code appears to be good. However, it contains a vulnerability. Upon closer inspection of the following code block:

return res.status(404).send(User with ID ${id} not found`);

In cases where the user is not found in the database, the code returns the searched-for ID in the message. This parameter can be manipulated, allowing for the execution of malicious code. For example, using the following payload:

localhost:3000?id=%3Cscript%3Ealert%28%22You%20was%20hacked%22%29%3C%2Fscript%3E%20

After making a call to the provided URL, this is the result:

In this scenario, we used a payload <script>alert(“you were hacked”)</script> that was encoded to be passed via URL. The result is the script running in the browser, which is not cool.

To prevent this issue, it's necessary to validate all input coming from the user. For example, in this case, the XSS attack could have been avoided if validation had been carried out on the ID property to ensure that it's a number. Additionally, it's important to avoid sending front-end received fields in the response, as this creates another barrier against XSS attacks.

One interesting detail to note is that if an error occurs and the compensating transaction generates a new event or updates the previous state, the order will not be deleted; instead, its status will be updated to "canceled."

So when you use the SAGA pattern, you meet with definitive success or definitive failure.

There are two approaches to implementing the SAGA pattern: choreography and orchestration.

Case 2: SQL Injection

Let's take a look at a SQL injection scenario.

In this case, ChatGPT was given the following directive:

Write a PHP endpoint that receives a user's ID and returns the retrieved user.

This is the code that ChatGPT provided:

At first glance, the code appears to be good and functional. However, it contains a well-known and dangerous vulnerability: SQL Injection.

The problem lies in this line of code:

"SELECT * FROM users WHERE id = " . $user_id;

Here, the ID passed in the request is concatenated directly into the SQL query. This is extremely dangerous, as it allows attackers to inject arbitrary queries into our database.

For example, an attacker could use the following payload:

localhost:3000?id=”1 OR 1=1;

This would generate the following query:

SELECT * FROM users WHERE id = 1 OR 1=1.

Since 1=1 is always true, this would bring back all users from the database. Not good, right?

To solve this problem, we must always validate user input. In this case, it is important to use parameterized queries when dealing with SQL queries. This prevents malicious queries from being executed in the database.

Keep this example in mind, as we will come back to it later to show you something interesting.

Case 3: Path Traversal

Now let's see an example where the code allows unauthorized access to files on the system.

Here, the following instruction was given to chatGPT:

"I have an endpoint that receives the name of a file in a URL parameter called `filename`."

"Create JavaScript code that reads the file with that name from the disk and returns it in the request."

The code returned the following output:

At first glance, this code may seem good, but like the previous ones, it has a vulnerability.

The vulnerability is Path Traversal or Directory Traversal. This occurs when an application allows user input to be used to build system paths, which can give an attacker access to files they should not have access to.

In this case, the vulnerability is found in this piece of code, where the argument filename is passed directly to the fs.readFile() function:

1. await fs.readFile(filename); 

1. chatGPT:
   - test: (Api's is currently running here)
   - assets:
     - file-which-the-user-can-read.txt
     - file-which-the-user-cannot-read.txt

By typing the following URL: localhost:3000/file/../../file-which-the-user-cannot-read.txt, we can access the document that the user should not have access to. Depending on the server configuration, it is possible to capture machine access data such as SSH keys.

To mitigate this risk, we must validate the name of the file we receive. For example, we can create a string with the value of the received file and check if the path formed is an allowed path. In addition, it is necessary to verify that the web server has access permissions properly configured. These measures can be effective against this type of attack.

Case 4: HPP (HTTP Parameter Pollution)

In this case, we will examine a flaw called HPP or HTTP Parameter Pollution in the code generated by ChatGPT. This flaw occurs when we pass more than one parameter to the application with the same name but with different values. It is often used to force errors and obtain information about the application.

The instruction we gave ChatGPT was to "write an endpoint in Node.js that receives a user ID via query parameters and fetches that user from the database."

Let's take a closer look at the code that ChatGPT provided:

Upon close inspection, we can see that the generated code has the aforementioned flaw in the following snippet:

const {id} = req.query;

If we call the URL "localhost:3000/user?id=5&id=6", the backend will receive an array with the values, like this "[5,6]". This can break your application because the code expects to receive a number but receives an array of numbers.

It is important to note that each runtime environment handles this situation differently, so it's crucial to have a good understanding of the technology you're working with. Here are the behaviors of the main programming languages/execution environments:

(font: https://www.imperva.com/learn/application-security/http-parameter-pollution/)

Hey, ChatGPT, did you know this all along?

After generating these codes with security flaws, I decided to do a test. Remember the SQL injection example we saw in PHP? So, without changing a comma, I asked ChatGPT, "Hey ChatGPT, is there any vulnerability in this code?" To my surprise (or not), this was the answer:

The code provided introduces a security vulnerability known as "SQL Injection". This vulnerability arises because the $user_id variable is defined using external user input, which in this case is obtained from the parameters of the GET request. An attacker can exploit this vulnerability by sending a GET request with a malicious entry for the id parameter. For instance, sending 1; DROP TABLE users; -- as the id parameter would cause the resulting SQL query to become: SELECT * FROM users WHERE id = 1; DROP TABLE users; --. This query would then execute a malicious code injection that would delete the entire "users" table from the database.

To fix this vulnerability, it is necessary to sanitize user input before using it in SQL query construction. There are several sanitization techniques that can be used to prevent SQL injection, such as PHP's mysqli_real_escape_string() function or the use of prepared statements. The important thing is not to trust external data without properly validating and sanitizing it.

Thus, the existence of this vulnerability proves that the code can generate incorrect and potentially destructive results.

Considerations

Looking at these failures, especially the last case where ChatGPT found security flaws in the code it generated, we can deduce that it currently does not verify the security or quality of the codes it generates.

This is understandable, as ChatGPT was created to answer general-purpose questions, not specific programming questions. Therefore, developers must keep in mind that it is indeed a great tool with numerous applications.

We cannot blindly trust the codes generated by ChatGPT or any other AI tool, as it was trained with data generated by humans, who are prone to making mistakes. This is especially important for new programmers, who tend to assume that the first version of working code they see is correct.

Therefore, I want to emphasize the need to be careful with the generated code, and to analyze and study it to check for any points of failure. This applies not only to code generated by ChatGPT, but also to any code copied from elsewhere.

It is important to make the software development process safer, and AI can be a great ally in this task. However, if used improperly, it can have the opposite effect. It all depends on the maturity and seriousness of those who use these tools.

Have you ever found a security flaw in code generated by ChatGPT or any other AI tool? Leave a comment and let us know!

That's all for today. I hope this tip is useful to you. Take care, stay safe, and see you next time!

References

"XSS Attack"

"HPP Attack"

"SQL Injection"

"Path Traversal"