San Francisco-based cryptocurrency exchange, Coinbase, has conducted a test on the utility of ChatGPT in conducting security analysis for ERC-20 tokens. Although the results did not meet the desired accuracy, Coinbase stated that the AI application showed potential.
Coinbase conducted a test using its own ERC-20 token review framework and compared the results of ChatGPT’s analysis with those of its internal team, which has internal tools to review blockchain projects.
Out of 20 smart contract risk scores, ChatGPT had the same result as manual reviews 60% of the time. However, in the eight incorrect outputs, the AI labeled a high-risk asset as a low-risk one five out of ten times.
Coinbase noted that ChatGPT is not capable of recognizing when it lacks context to perform a robust security analysis and is prone to inconsistency. Upgrades to the AI may also result in output instability.
Coinbase’s previous experimentation with ChatGPT
This is not the first time that Coinbase has experimented with ChatGPT. Earlier this month, Coinbase’s Conor Grogan stated that ChatGPT was successful in finding bugs in smart contracts, offering a specific means to exploit the contract.
I dumped a live Ethereum contract into GPT-4.
In an instant, it highlighted a number of security vulnerabilities and pointed out surface areas where the contract could be exploited. It then verified a specific way I could exploit the contract pic.twitter.com/its5puakUW
— Conor (@jconorgrogan) March 14, 2023
As the newly released of ChatGPT-4 model continues to generate buzz in the tech world, developers of Ethereum have stated that they do not believe the application will replace developers. The mania surrounding ChatGPT has also sparked a rally in AI-based tokens in the crypto market, but users must exercise caution as some may be scams.
In conclusion, although ChatGPT did not hit the desired accuracy in its test with Coinbase, it showed potential in the application of AI in the crypto industry. As the technology continues to advance, it will be interesting to see how it will shape the future of blockchain security analysis.