AI code assistants observations
Output based on quality of input
2
Fickleness on optimized solutions
6
Questionable outcomes
14
Unreliable code reviewer
20
Lax approach towards licensed code
27
Output based on quality of input
There may be multiple solutions to a problem, and sometimes AI generative tools may not
suggest the desired solution. It is important to be specific about your requirements in order to
increase the chances of finding a suitable solution. Here are a few examples:
Question 1
I want to make an API call with latest parameters and if user tries to call the API again before
the response, cancel the previous call and call new API with latest parameters. Write this code
in Angular way
Answer 1 (Provided the answer with approach I was not looking for)
Question 1.1 (I asked a more specific question)
Is it not possible with above solution with SwitchMap operator?
Answer 1.1(More Specific):
Question 2 - Write a java program to merge n sorted arrays
Answer 2
Question 2.1 (Asked the question to follow the specific approach)
Write the same program without priorityqueue
Answer 2.1
Fickleness on optimized solutions
The use of these tools does not necessarily guarantee the most optimized solution on the first
try. Without a sufficient level of knowledge, a developer may not consider alternative solutions
and may not ask the tool to provide a better result.
Besides this, the result may not provide the necessary null check and boundary conditions
check that a developer must consider.
Question 1
find the summation of the Fibonacci series up to the 20th number
Answer 1 (The approach the ChatGPT provided is with for loop)
Question 1.1
Can we rewrite the above logic with recursion? (We need to provide instruction to follow a better
convention and then ChatGPT was able to answer it accordingly.)
Answer 1.1 (Answer with recursion)
Question 2
Can you rewrite the below code
function getDrink (type) {
if (type === 'coke') {
type = 'Coke';
} else if (type === 'pepsi') {
type = 'Pepsi';
} else if (type === 'mountain dew') {
type = 'Mountain Dew';
} else if (type === 'lemonade') {
type = 'Lemonade';
} else if (type === 'fanta') {
type = 'Fanta';
} else {
// acts as our "default"
type = 'Unknown drink!';
}
return 'You\'ve picked a ' + type;
}
Answer 2 (Provide the default answer using Switch case)
Now we asked ChatGPT to follow the better approach using object literals and then it provided
the response accordingly.
Question 2.1 (Question to follow better approach )
Can you rewrite the above code with object literals approach?
Answer 2.1 (More specific as per instruction given)
Question 3
Write a java program to convert ipv4 address to a unique integer
Answer 3
Question 3.1 (unit test this code for 192.168.1)
Now we asked ChatGPT to add a unit test case for the generated code, and we provided
incorrect IP address input to validate the boundary condition check in the code, and we received
a response from the bot addressing the missing boundary condition check. Here's the response:
Answer 3.1
Question 3.2 (please modify the code for handling such scenarios)
Now we asked the bot to correct the code with proper boundary condition checks, and it
corrected the code with instructions that was not done in the first place.Here is the response in
accordance with the instructions:
Answer 3.2
Questionable outcomes
AI chatbots may sometimes provide responses with confidence, but they may not always be
correct. It is important to validate their results with our own experience to ensure their accuracy.
For example:
Question 1
●
Can you rewrite this code with better approach
function findFizz(num) {
if (num % === 0) {
return "Jaydeep";
} else if (num % 15 === 0) {
return "Mayank";
} else if (num % 5 === 0) {
return "Ritesh";
} else {
return number;
}
}
Answer 1:
Question 1.1 (We asked for another approach)
Now I think we should try object literals approach
Answer 1.1 (The bot's provided solution is incorrect and it modified the conditions provided in
original code)
Question 1.2 (Asking AI assistant to validate the wrong answer)
Do you think this object literals solution is same as if statement solution?
Answer 1.2 (Bot confirms the wrong answer with confidence)
Let’s take another example:
Question 2
what is the probablistic data structure in Java that ensures a collection is sorted at all times
Answer 2: (Bot is saying there is no such probabilistic data structure with confidence)
Question 2.1(Giving specific direction to think again for the solution)
can a skippedlist be used to maintain a collection that is sorted at all times?
Answer 2.1 (Previously it answered “there is no probabilistic data structure”)
Question 3
Write a java program that computes the time complexity of searching in a LinkedList
Answer 3 (The question was to compute the time complexity in O notation. However, the
program simply calculates the time it takes to search an element in a list.)
Unreliable code reviewer
Question 1
Can you help me to review this code
var target = { name: "xyz", age: 20 };
var source1 = { name: "abc", grade: 12 };
var source2 = { gender: "female" };
updatedTarget = Object.assign(target, source1, source2);
console.log(updatedTarget, target);
Answer 1
It implies that the code is correct, but it could have been written better using the ES6 object
literals syntax.
Question 1.1 (Giving hint to check for better approach)
is there a better coding convention?
Answer 1.1 (Suggested better approach after given hint for code review)
(Instead of object.assign, ChatGPT should have tried the approach of spead operator, which is
a better way to solve the problem. It corrected after being asked to provide better coding
conventions.) Here is the response:
Question 2
Can you review this code in Java
/ Online Java Compiler
// Use this editor to write, compile and run your Java code online
class Employee {
private String id;
private String name;
private String address;
private Double salary;
public Employee(String id, String name, String address, Double salary) {
this.id = id;
this.name = name;
this.address = address;
this.salary = salary;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getAddress() {
return address;
}
public void setAddress(String address) {
this.address = address;
}
public Double getSalary() {
return salary;
}
public void setSalary(Double salary) {
this.salary = salary;
}
public Employee(String id) {
this.id = id;
}
}
class HelloWorld {
public static void main(String[] args) {
Employee e1 = new Employee("111");
Employee e2 = new Employee("112");
Map h = new HashMap();
h.put(e1, e1);
h.put(e2, e2);
e2.setId("113");
}
}
In the above code there are several issues related to using Employee class as key in the
hashmap. According to the best practices the Employee class should be immutable. ChatGPT
was able to mention the basic rules of using user-defined class as key such as overriding
hashCode and equals methods. However it did not point out that Employee class is not
immutable. In the code we actually tried to mutate one of the employee objects which is used as
key in the HashMap. ChatGPT did not point out this mutation as a problem which is a serious
problem. An experienced developer would catch this problem in first place.
Here is the code review comments given by ChatGPT which didn’t mention the above point:
Answer 2
Question 3
Write a JS code for submitting a form including name, email address, age, gender, postal
address
Answer 3 ( The below code contains only basic validations like null or empty inputs but does not put in
necessary checks for SQL injection or XSS attacks. )
Lax approach towards licensed code
ChatGPT will provide you code from commercially licensed libraries without warning. Unless you
specifically request that the code be used for commercial purposes, it does not alert you of the
legal consequences.
Question 1
Please provide Pie chart from FusionCharts library in javascript
Answer 1 (Provided the answer for a commercial license library without mentioning about
commercial license)
Question 1.1
Can I directly use this code in my commercial application?
Answer 1.1 (with specific question, ChatGPT was able to provide the answer)
Now I'm telling ChatGPT about the code I want to use in a commercial application.
Question 2 (Asking specific question with code for commercial application)
I want to use highcharts line graph in my commercial project. Please suggest me the code
Answer 2
ChatGPT not only provides the result, but it also provides guidelines in the result about
commercial licenses associated with specific library.