Comparing the Effectiveness of Bard, Perplexity, and ChatGPT-4 in Generating Investment Strategies for SmartBots

This article aims to compare the effectiveness of three artificial intelligence chat assistants — Bard, Perplexity, and ChatGPT-4 — in generating investment strategies. The research focuses on evaluating the applicability of strategies generated for SmartBots, a Python-based trading platform. Using specific criteria, this study examines the consistency, clarity, innovation, and effectiveness of the strategies proposed by each chatbot. Furthermore, a deep comparative analysis of responses to a set of financial investment-related questions is conducted. The ultimate goal is to understand how these AI tools could be utilized to enhance decision-making in the investment world, especially within the context of an automated trading platform like SmartBots

Introduction:

Why is it relevant to compare AI assistants in the investment field?

The comparison of AI assistants in investment is essential given their growing influence in automating and improving financial analysis. These assistants democratize access to advanced insights and tailor responses according to the investor’s needs. Identifying which assistant integrates best with trading platforms, like SmartBots, is crucial for offering advanced tools to users.

Brief description of SmartBots:

SmartBots is a Python-developed trading platform that eases the implementation of strategies from idea to real-time execution. It connects various technologies, offering backtesting, real-time alerts, and an event-driven architecture. It is adaptable to any financial market, from cryptocurrencies to sports betting, more info here

Methodology:

The three chatbots (Bard, Perplexity, and ChatGPT-4) will be judged using a common method. The following steps will be followed:

Prompt Selection: A set of investment-related questions or scenarios will be designed. This set will be identical for each chatbot to ensure consistency in comparisons.

Interaction and Recording: The same prompt will be presented to each chatbot**.**

Implementation on SmartBots: The chatbots’ generated strategies will be implemented on the SmartBots platform.

Results Analysis: Responses will be evaluated for consistency, innovation, effectiveness, and other relevant criteria. Failures or inefficiencies will also be identified and analyzed.

Documentation and GitHub: To keep the article concise and accessible, detailed information and specific results will be uploaded to a GitHub repository. A link will be provided in the article for those interested in technical details.

Description of the Chatbots:

Bard:

Bard is an AI system developed by Google, based on its LaMDA language model. Designed for natural conversations, Bard offers real-time responses, drawing information directly from the Internet. Unlike other chatbots, Bard provides updated information as it is integrated with Google’s search engine and is designed for various applications.

ChatGPT-4:

Developed by OpenAI, ChatGPT-4 is a language model with over 175 million parameters. It is renowned for generating well-formed and contextualized responses across a wide range of topics, although it does not have real-time internet access.

Perplexity-Claude-2

Perplexity AI is a conversational answer engine that utilizes CHATGPT-4 and Claude-2 to deliver precise responses to complex questions. In this example, we will be selecting Claude-2. Perplexity stands out for its ability to search the vast expanse of the internet, providing real-time answers and sources for further verification.

Generation of Investment Strategies:

Trading Strategy Prompt:

YOUR REQUEST: “I want a strategy that applies the Moving Average Convergence/Divergence (MACD). If the short-period MACD is greater than the long-period MACD, it should send a buy order, and If the long-period MACD is greater than the short-period MACD, it should send a sell order.”

You are a specialized Python coding assistant for quantitative trading with trading systems and strategies. Using the following 18 guidelines, 
create a new class that inherits from the Abstract_Strategy class and implement the next request:
  """ YOUR REQUEST """

1. Import the Abstract_Strategy class from **`src.domain.abstractions.abstract_strategy`**
2. Create a new class that inherits from Abstract_Strategy.
3. Use methods and functions to encapsulate the calculation of any indicator and algorithm from the request.
4. Override the add_event method and implement the algorithms and indicators from the request.
5. Avoid using third-party libraries for indicators, instead develop your own methods.
6. Do not use callbacks in the add_event method.
7. Include detailed comments in the code to explain the methods, functions and classes.
8. Use the self.ticket, price, self.quantity, action, type, and datetime arguments when calling the send_order method.
9. Event received in the add_event method is a dataclass type, with attributes for close, open, high, low, volume, and ticker.
10. Provide a dictionary of inputs for any indicator you create from the request in the `parameters` argument from init method.
11. When conditions for buying and selling are met, use the send_order method with keyword arguments.
12. When conditions for buying and selling are met, only send one order until the strategy change.
13. Use event.event_type checking, it can be ‘bar’ and ‘tick’ type.
14. Use self.position and do not change value of self.position in add_event method. send_order method calculate self.number_of_contracts variable that you have to use in send_order implementation.
15. Preallocate memory to avoid storing more data than necessary with variables. 
16. Refer to past events to calculate variable if the request is not expressed in another way.
17. Use _add_event_example as an example for implement add_event.
18. Create an answer within one code block.

Context: """ Copy Code from <https://github.com/SmartLever/SmartBots/blob/main/src/domain/abstractions/abstract_strategy.py> """

We presented the same prompt to Bard, ChatGPT-4, and Perplexity to generate strategies. The refined and corrected versions of these strategies can be found in our GitHub repository. They are named as follows:

• Bard: macd_bard
• ChatGPT-4: macd_chatgpt4
• Perplexity: macd_perplexity

You can explore them in detail here

Analysis of the Responses:

In this section, errors and corrections made for each chatbot are broken down. Each sub-section deals with a specific chatbot, highlighting the challenges and the solutions applied.

ChatGPT-4:

Challenges:

1. Lacked default parameter values, affecting usability.
2. Did not update self.position to prevent multiple identical transactions.
3. Failed to update equity during close_day events.
4. Lacked the set_basic argument in both __init__ and super().

Solutions Applied:

1. Added reasonable default values for parameters based on empirical testing or financial literature.
2. Updated self.position within the send_order method to prevent duplicate orders.
3. Included equity update logic when a close_day event is received.
4. Included the set_basic argument in both __init__ and super() for code coherence.

For Chatgpt4 original version:

from src.domain.abstractions.abstract_strategy import Abstract_Strategy, Order
from dataclasses import dataclass
import datetime as dt

class MACDStrategy(Abstract_Strategy):
    """
    MACD Strategy implementation.
    Inherits from Abstract_Strategy class.
    """

    def __init__(self, parameters: dict, id_strategy: int = None, callback: callable = None):
        """
        Constructor to initialize MACDStrategy class.
        
        :param parameters: Dictionary containing parameters for the strategy.
        :param id_strategy: Unique identifier for the strategy.
        :param callback: Callback function.
        """
        # Initialize the base class
        super().__init__(parameters, id_strategy, callback)
        
        # Initialize additional parameters specific to MACD
        self.short_window = parameters['short_window']
        self.long_window = parameters['long_window']
        self.signal_window = parameters['signal_window']
        
        # Preallocate memory for storing past events
        self.close_prices = []
        
    def calculate_MACD(self, close_prices: list) -> float:
        """
        Method to calculate MACD
        
        :param close_prices: List of close prices.
        :return: MACD value.
        """
        short_avg = sum(close_prices[-self.short_window:]) / self.short_window
        long_avg = sum(close_prices[-self.long_window:]) / self.long_window
        return short_avg - long_avg
    
    def add_event(self, event: dataclass):
        """
        Override the add_event method to implement MACD strategy.
        
        :param event: Event data.
        """
        if event.event_type == 'bar':
            # Update the internal state
            self.close_prices.append(event.close)
            
            # Ensure we have enough data to calculate MACD
            if len(self.close_prices) >= self.long_window:
                
                # Calculate MACD
                macd_value = self.calculate_MACD(self.close_prices)
                
                # Send orders based on MACD
                if macd_value > 0 and self.position <= 0:  # MACD above zero and no long position
                    self.send_order(price=event.close, quantity=self.quantity, action='buy', 
                                    type='market', datetime=event.datetime, ticker=self.ticker)
                    
                elif macd_value < 0 and self.position >= 0:  # MACD below zero and no short position
                    self.send_order(price=event.close, quantity=self.quantity, action='sell', 
                                    type='market', datetime=event.datetime, ticker=self.ticker)
        
        elif event.event_type == 'tick':
            # Additional logic can be added here for 'tick' type events
            pass

Perplexity with Claude-2:

Challenges:

1. Missing parameters id_strategy, callback, and set_basic in both __init__ and super().
2. Did not initialize self.closes as an empty list.
3. Lacked differentiation between event types in add_event.
4. Incorrect MACD calculation.

Solutions Applied:

1. Included id_strategy, callback, and set_basic in both __init__ and super().
2. Initialized self.closes as an empty list in the constructor.
3. Included logic to differentiate between event types in add_event.
4. Referred to ChatGPT-4 for a correct MACD calculation.

For Perplexity’s original version, visit here

Bard:

Challenges:

1. Did not import dataclass.
2. Lacked event type differentiation in the add_event method.
3. Failed to update equity during close_day events.
4. Incorrect MACD calculation.

Solutions Applied:

1. Imported dataclass for type annotation.
2. Differentiated between ‘bar’ and ‘tick’ events in the add_event method.
3. Updated equity during close_day events.
4. Referred to ChatGPT-4’s MACD calculation for a correct implementation.

For Bard’s original version, visit here

Applicability in SmartBots:

To validate the effectiveness of the strategy generated by one of the chatbots, we conduct a simulation using the SmartBots platform. Within this Notebook, users can easily configure the parameters of the strategy and select the desired time frame for the simulation. The notebook then executes the strategy, producing detailed graphical outputs and key performance metrics. These screenshots and figures will be included to offer a tangible demonstration of the strategy’s feasibility and potential returns, reinforcing its applicability in real-world scenarios. For those interested in a deeper dive, the entire simulation notebook is accessible here.

Simulating:

Stats:

Conclusive Remarks:

This study unequivocally demonstrates that ChatGPT-4 stands out as the most reliable and effective artificial intelligence assistant for generating investment strategies on the SmartBots platform. With minimal corrections needed, it offers a viable pathway for automated implementation of investment strategies, setting a high bar for future AI implementations in the financial field.

In contrast, while Bard and Perplexity show potential, their current limitations hinder their immediate applicability for efficient and effective investment strategies. However, with further improvements and adaptations, these systems could eventually close the gap.

Therefore, for SmartBots users and others interested in the application of AI to financial investing, ChatGPT-4 currently stands as the most robust and reliable option.

The train is in the station, and you can’t miss it, so join us on this journey. We are an open community and if you want to collaborate with us, please send us an email: andres@smartlever.tech