Trading Robot Based on a GPT Language Model

Introduction

TimeGPT is a machine learning model developed to predict price movements in financial markets, such as forex pairs or stocks, on the MetaTrader 5 platform. It is based on the Transformer architecture, which was originally designed for processing text, such as translation or text generation, but is now adapted for financial data analysis. In this article, I will explain in detail how TimeGPT is created step by step, covering its architecture, data processing, training, and forecasting. Everything will be explained in simple language, with analogies, so that even someone just beginning to study machine learning can understand how such a model is created. Finally, I will also touch on the limitations of the model and its computational requirements.

What does it take to create a model like TimeGPT?

Before diving into the code, it is important to understand what TimeGPT works with. Financial markets produce data in the form of time series — a sequence of prices recorded at regular intervals, such as every hour. Market prices do not follow simple rules: they can skyrocket on news, slowly fall, or fluctuate for no apparent reason. TimeGPT's job is to find patterns in this data and predict how the price will change in 24 hours, which in market charting terms is called 24 bars.

Developing a model requires solving several problems. First, we need to determine how to store and process price data. Then we need to convert this data into a format the model can understand. Next, we need to design the model architecture so that it can find complex dependencies in the data. The model is then trained on historical data to learn how to make accurate predictions. Finally, it is important to optimize the model so that it runs quickly and does not require too much memory, given the limitations of the MetaTrader 5 platform. Now let's look at each of these steps, using the code from the TimeGPT_Fixed.mqh file.

Model development

Creating a structure for working with data

The first step in developing TimeGPT was to create a structure for storing and processing data. In machine learning, data is often represented as matrices — tables of numbers that are convenient for mathematical operations. For this purpose, the TGMatrix structure was created in TimeGPT, which allows storing numbers such as prices or model parameters in an ordered form.

Here is what the code for this structure looks like:

struct TGMatrix {
    double data[];
    int rows, cols;
    
    void Init(int r, int c) {
        rows = r; 
        cols = c;
        ArrayResize(data, r * c);
        ArrayInitialize(data, 0.0);
    }
    
    double Get(int r, int c) const { 
        if(r < 0 || r >= rows || c < 0 || c >= cols) return 0.0;
        return data[r * cols + c]; 
    }
    
    void Set(int r, int c, double val) { 
        if(r < 0 || r >= rows || c < 0 || c >= cols) return;
        data[r * cols + c] = val; 
    }
    
    void RandomInit(double scale = -1.0) {
        if(scale < 0) scale = MathSqrt(2.0 / (rows + cols));
        for(int i = 0; i < ArraySize(data); i++) {
            data[i] = (MathRand() / 32767.0 - 0.5) * 2.0 * scale;
        }
    }
    
    void Add(const TGMatrix &other) {
        if(rows != other.rows || cols != other.cols) return;
        for(int i = 0; i < ArraySize(data); i++) {
            data[i] += other.data[i];
        }
    }
    
    void Copy(const TGMatrix &other) {
        Init(other.rows, other.cols);
        for(int i = 0; i < ArraySize(other.data); i++) {
            data[i] = other.data[i];
        }
    }
    
    void Scale(double factor) {
        for(int i = 0; i < ArraySize(data); i++) {
            data[i] *= factor;
        }
    }
};

Think of TGMatrix as an Excel table where numbers are stored in rows and columns. The Init method creates a table of the required size, Get and Set allow you to read and write values to specific cells, RandomInit fills the table with random numbers that are needed for the initial setup of the model, and the Add, Copy, and Scale methods perform basic operations such as adding two tables or multiplying all the numbers by some value. This structure became the basis for working with data in TimeGPT, because all prices, model parameters and calculations use matrices. We chose this approach because it is simple and works well in MetaTrader 5, where complex libraries like Python cannot be used.

Converting prices into a format understandable to the model

In order for the model to analyze prices, they need to be converted into a convenient format. TimeGPT borrows an idea from text processing: every price change is turned into a "token" — a number that represents the price movement, such as a 0.5% increase or a 0.3% decrease. The ImprovedTokenizer structure is responsible for that.

Here is its code:

struct ImprovedTokenizer {
    double price_mean, price_std;
    double change_mean, change_std;
    bool is_trained;
    
    ImprovedTokenizer() {
        is_trained = false;
        price_mean = 0; price_std = 1;
        change_mean = 0; change_std = 1;
    }
    
    void Train(const MqlRates &rates[]) {
        int size = ArraySize(rates);
        if(size < FORECAST_HORIZON + 10) return;
        
        // Calculate statistics for prices
        double sum_price = 0, sum_price_sq = 0;
        double changes[];
        ArrayResize(changes, size - FORECAST_HORIZON);
        
        for(int i = FORECAST_HORIZON; i < size; i++) {
            double price = rates[i].close;
            sum_price += price;
            sum_price_sq += price * price;
            
            // Price change after 24 bars
            changes[i - FORECAST_HORIZON] = (rates[i].close - rates[i - FORECAST_HORIZON].close) / rates[i - FORECAST_HORIZON].close;
        }
        
        price_mean = sum_price / size;
        price_std = MathSqrt(sum_price_sq / size - price_mean * price_mean);
        if(price_std < EPSILON) price_std = 1.0;
        
        // Statistics for changes over the next 24 bars
        double sum_change = 0, sum_change_sq = 0;
        for(int i = 0; i < ArraySize(changes); i++) {
            sum_change += changes[i];
            sum_change_sq += changes[i] * changes[i];
        }
        
        change_mean = sum_change / ArraySize(changes);
        change_std = MathSqrt(sum_change_sq / ArraySize(changes) - change_mean * change_mean);
        if(change_std < EPSILON) change_std = 1.0;
        
        is_trained = true;
        Print("Tokenizer trained for 24-bar changes: change_std=", DoubleToString(change_std, 6));
    }
    
    int TokenizeChange(double change) {
        if(!is_trained) return VOCAB_SIZE / 2;
        
        // Normalize and quantize
        double normalized = (change - change_mean) / change_std;
        normalized = MathMax(-3.0, MathMin(3.0, normalized)); // Clamp
        
        int token = (int)((normalized + 3.0) / 6.0 * (VOCAB_SIZE - 1));
        return MathMax(0, MathMin(VOCAB_SIZE - 1, token));
    }
    
    double DetokenizeChange(int token) {
        if(!is_trained || token < 0 || token >= VOCAB_SIZE) return 0.0;
        
        double normalized = (double)token / (VOCAB_SIZE - 1) * 6.0 - 3.0;
        return normalized * change_std + change_mean;
    }
};

The tokenizer works as follows. The Train method analyzes historical candle closing prices, for example, over the last 2000 hours, and calculates how they have changed over 24 bars. If the price was USD 100 and after 24 hours it became USD 102, the change would be (102 - 100) / 100 = 2%. The tokenizer then calculates the mean and standard deviation of all changes to bring them to a uniform scale. It is like adjusting a ruler so that all changes are measured in the same units. The TokenizeChange method turns each change into a number between 0 and 255 (vocabulary size VOCAB_SIZE = 256), for example, a 2% increase might become the token "150". The DetokenizeChange method performs the reverse transformation indicating which price change corresponds to a token.

This approach is important because raw prices can vary greatly: one stock is worth USD 10, while another is worth USD 1,000. Tokenization makes the data simpler for the model, allowing it to work with numbers in a single range, just like text, where each word has its own code.

Building an attention mechanism

The key part of TimeGPT is the attention mechanism implemented in the SimpleAttention structure. It allows the model to focus on important price changes in the past while ignoring less significant ones. Imagine reading a long book and highlighting only the key points that help you understand the plot. The attention mechanism does the same thing, deciding which price data is important for the forecast.

Here is the code for the attention mechanism:

struct SimpleAttention {
    TGMatrix W_q, W_k, W_v, W_o;
    double scale;
    
    void Init() {
        W_q.Init(MODEL_DIM, MODEL_DIM);
        W_k.Init(MODEL_DIM, MODEL_DIM);
        W_v.Init(MODEL_DIM, MODEL_DIM);
        W_o.Init(MODEL_DIM, MODEL_DIM);
        
        double init_scale = MathSqrt(2.0 / MODEL_DIM);
        W_q.RandomInit(init_scale);
        W_k.RandomInit(init_scale);
        W_v.RandomInit(init_scale);
        W_o.RandomInit(init_scale);
        
        scale = 1.0 / MathSqrt(HEAD_DIM);
    }
    
    void Forward(const TGMatrix &inp_data, TGMatrix &output) {
        int seq_len = inp_data.rows;
        output.Init(seq_len, MODEL_DIM);
        
        // Multi-headed attention
        for(int head = 0; head < NUM_HEADS; head++) {
            int head_start = head * HEAD_DIM;
            
            // Simple projections for this head
            TGMatrix head_output;
            head_output.Init(seq_len, HEAD_DIM);
            
            for(int i = 0; i < seq_len; i++) {
                // Calculate attention only to previous positions
                double total_weight = 0.0;
                double weighted_values[HEAD_DIM];
                ArrayInitialize(weighted_values, 0.0);
                
                for(int j = 0; j <= i; j++) {
                    // Simple dot product for attention weight
                    double attention_weight = 0.0;
                    for(int k = head_start; k < head_start + HEAD_DIM; k++) {
                        attention_weight += inp_data.Get(i, k) * inp_data.Get(j, k);
                    }
                    attention_weight = MathExp(attention_weight * scale);
                    total_weight += attention_weight;
                    
                    // Accumulate weighted values
                    for(int k = 0; k < HEAD_DIM; k++) {
                        weighted_values[k] += attention_weight * inp_data.Get(j, head_start + k);
                    }
                }
                
                // Normalize and write
                if(total_weight > EPSILON) {
                    for(int k = 0; k < HEAD_DIM; k++) {
                        head_output.Set(i, k, weighted_values[k] / total_weight);
                    }
                }
            }
            
            // Copy the head result to the output matrix
            for(int i = 0; i < seq_len; i++) {
                for(int j = 0; j < HEAD_DIM; j++) {
                    output.Set(i, head_start + j, head_output.Get(i, j));
                }
            }
        }
    }
};

The attention mechanism divides the data into six parts called "heads" (NUM_HEADS = 6), each of which looks for different patterns, such as short-term price spikes or long-term trends. The Init method creates W_q, W_k, W_v, and W_o matrices used to transform the data, and fills them with random numbers. In the Forward method, the model calculates how important each previous position in the sequence is to the current one using the dot product and the MathExp function.

It is important that the model only looks at the data up to the current position (j <= i), so as not to "peek" into the future - this is called causal masking. The attention weights are normalized so that they sum to one and are used to weight the data, as if the model were saying, "This moment is 70% important, and that moment is only 20% important".

This mechanism allows TimeGPT to find complex relationships between price changes, even if they occurred long ago, making it more powerful than older models that quickly "forget" past data.

Assembling Transformer layers

TimeGPT consists of six layers (NUM_LAYERS = 6), each of which includes an attention mechanism and additional data processing via a feed-forward network (FFN). It is like a filter that gradually improves the understanding of the data by passing through several stages of processing.

Here is the code for one layer:

struct TransformerLayer {
    SimpleAttention attention;
    SimpleFeedForward ffn;
    
    void Init() {
        attention.Init();
        ffn.Init();
    }
    
    void Forward(const TGMatrix &inp_data, TGMatrix &output) {
        // Attention with residual connection
        TGMatrix attn_out;
        attention.Forward(inp_data, attn_out);
        
        TGMatrix residual1;
        residual1.Copy(inp_data);
        residual1.Add(attn_out);
        
        // FFN with residual connection
        TGMatrix ffn_out;
        ffn.Forward(residual1, ffn_out);
        
        output.Copy(residual1);
        output.Add(ffn_out);
    }
};

Each layer first applies an attention mechanism to highlight important parts of the data, then adds the original data to the result (this is called residual connection) to avoid losing information. The data is then passed through a feed-forward network, which adds nonlinearity, allowing for the discovery of complex patterns. Another residual connection adds the FFN result to the data after attention, maintaining stability.

This is what FFN looks like:

struct SimpleFeedForward {
    TGMatrix W1, W2, b1, b2;
    
    void Init() {
        W1.Init(MODEL_DIM, FFN_DIM);
        W2.Init(FFN_DIM, MODEL_DIM);
        b1.Init(1, FFN_DIM);
        b2.Init(1, MODEL_DIM);
        
        double init_scale = MathSqrt(2.0 / MODEL_DIM);
        W1.RandomInit(init_scale);
        W2.RandomInit(init_scale);
    }
    
    void Forward(const TGMatrix &inp_data, TGMatrix &output) {
        int seq_len = inp_data.rows;
        output.Init(seq_len, MODEL_DIM);
        
        for(int i = 0; i < seq_len; i++) {
            // First layer with ReLU
            double hidden[FFN_DIM];
            for(int j = 0; j < FFN_DIM; j++) {
                hidden[j] = b1.Get(0, j);
                for(int k = 0; k < MODEL_DIM; k++) {
                    hidden[j] += inp_data.Get(i, k) * W1.Get(k, j);
                }
                hidden[j] = MathMax(0.0, hidden[j]); // ReLU
            }
            
            // Second layer
            for(int j = 0; j < MODEL_DIM; j++) {
                double sum = b2.Get(0, j);
                for(int k = 0; k < FFN_DIM; k++) {
                    sum += hidden[k] * W2.Get(k, j);
                }
                output.Set(i, j, sum);
            }
        }
    }
};

FFN takes the data and transforms it through two layers: the first one expands the data to a size of FFN_DIM = 256 and applies a ReLU function that zeros out negative values, adding nonlinearity, and the second one compresses the data back to MODEL_DIM = 256. This helps the model detect complex combinations of price changes that are not visible to simple linear methods.

Transformer layers are the basis of the model. Each layer enhances your understanding of the data, and six layers allow you to capture increasingly complex patterns. Residual connections make learning stable by preventing information loss.

Combining components into the TimeGPT model

Once all the parts are created, we will assemble them into a single FixedTimeGPT model. Here is its structure:

struct FixedTimeGPT {
    ImprovedTokenizer tokenizer;
    TGMatrix embeddings;
    TransformerLayer layers[NUM_LAYERS];
    TGMatrix output_projection;
    double learning_rate;
    int training_steps;
    bool is_trained;
    
    void Init() {
        embeddings.Init(VOCAB_SIZE, MODEL_DIM);
        embeddings.RandomInit(0.1);
        
        for(int i = 0; i < NUM_LAYERS; i++) {
            layers[i].Init();
        }
        
        output_projection.Init(MODEL_DIM, VOCAB_SIZE);
        output_projection.RandomInit(0.1);
        
        learning_rate = LEARNING_RATE;
        training_steps = 0;
        is_trained = false;
        
        Print("Fixed TimeGPT initialized for 24-bar forecast");
    }
};

This structure unites all components. The tokenizer converts prices into tokens. The embeddings matrix turns each token into a 256-valued vector of numbers so that the model can process it similarly to how language models represent words. Transformer's six layers process data to find patterns. The output_projection matrix transforms the output of the last layer into probabilities for each token (from 0 to 255) to predict the next price change. The learning_rate parameter determines how fast the model learns, and training_steps tracks the number of training steps.

This structure is simple enough to work on MetaTrader 5, but still powerful enough to find complex patterns.

Model training

Training is the process in which a model analyzes historical data and adjusts its parameters to make accurate predictions. Here is how it is implemented:

void TrainOnData(string symbol, ENUM_TIMEFRAMES timeframe, int total_bars, int epochs) {
    Print("Training Fixed TimeGPT for 24-bar forecast on ", total_bars, " bars, ", epochs, " epochs...");
    
    // Get data
    MqlRates rates[];
    ArraySetAsSeries(rates, false); // Chronological order
    if(CopyRates(symbol, timeframe, 0, total_bars, rates) <= 0) {
        Print("Failed to copy rates");
        return;
    }
    
    // Train the tokenizer for changes every 24 bars
    tokenizer.Train(rates);
    
    // Prepare sequences for training
    int num_sequences = total_bars - CONTEXT_LENGTH - FORECAST_HORIZON - 1;
    if(num_sequences <= 0) {
        Print("Not enough data for training");
        return;
    }
    
    // Learning cycle
    for(int epoch = 0; epoch < epochs; epoch++) {
        Print("Epoch ", epoch + 1, "/", epochs);
        double total_loss = 0.0;
        int batch_count = 0;
        
        // Handle sequences
        for(int start = 0; start < num_sequences; start += CONTEXT_LENGTH/2) {
            if(start + CONTEXT_LENGTH + FORECAST_HORIZON >= ArraySize(rates)) break;
            
            double loss = TrainStep(rates, start);
            total_loss += loss;
            batch_count++;
            training_steps++;
            
            // Decay learning rate
            if(training_steps % 100 == 0) {
                learning_rate *= 0.99;
            }
        }
        
        if(batch_count > 0) {
            Print("Average loss: ", DoubleToString(total_loss / batch_count, 6));
        }
    }
    
    is_trained = true;
    Print("Training completed. Total steps: ", training_steps);
}

The model loads historical prices, for example, for 2000 bars, for a currency pair such as EURUSD on an hourly chart. The tokenizer analyzes the data to understand typical price changes. The model then runs through the data several times (three epochs), each time taking a sequence of 64 bars (CONTEXT_LENGTH = 64) and trying to predict the price change 24 bars later.

The TrainStep method calculates the error (loss) — the difference between the forecast and the actual change — and adjusts the model parameters to reduce this error. Every 100 steps, the learning rate is reduced by 1% to make the model learn more carefully and avoid large errors.

This is what TrainStep looks like:

double TrainStep(const MqlRates &rates[], int start_pos) {
    // Simple learning step for forecasting in 24 bars
    int tokens[CONTEXT_LENGTH + 1];
    
    for(int i = 0; i < CONTEXT_LENGTH; i++) {
        int idx = start_pos + i;
        if(idx > 0 && idx < ArraySize(rates)) {
            double change = (rates[idx].close - rates[idx-1].close) / rates[idx-1].close;
            tokens[i] = tokenizer.TokenizeChange(change);
        } else {
            tokens[i] = VOCAB_SIZE / 2;
        }
    }
    
    // Target - change in 24 bars
    int target_idx = start_pos + CONTEXT_LENGTH + FORECAST_HORIZON;
    if(target_idx < ArraySize(rates) && start_pos + CONTEXT_LENGTH > 0) {
        double change = (rates[target_idx].close - rates[target_idx - FORECAST_HORIZON].close) / rates[target_idx - FORECAST_HORIZON].close;
        tokens[CONTEXT_LENGTH] = tokenizer.TokenizeChange(change);
    } else {
        tokens[CONTEXT_LENGTH] = VOCAB_SIZE / 2;
    }
    
    // Forward pass
    TGMatrix embedded;
    embedded.Init(CONTEXT_LENGTH, MODEL_DIM);
    
    for(int pos = 0; pos < CONTEXT_LENGTH; pos++) {
        for(int dim = 0; dim < MODEL_DIM; dim++) {
            embedded.Set(pos, dim, embeddings.Get(tokens[pos], dim));
        }
    }
    
    TGMatrix current;
    current.Copy(embedded);
    
    for(int i = 0; i < NUM_LAYERS; i++) {
        TGMatrix layer_out;
        layers[i].Forward(current, layer_out);
        current.Copy(layer_out);
    }
    
    // Simple loss
    int target = tokens[CONTEXT_LENGTH];
    double loss = 0.0;
    
    for(int i = 0; i < MODEL_DIM; i++) {
        double pred = current.Get(CONTEXT_LENGTH - 1, i);
        double target_emb = embeddings.Get(target, i);
        double diff = pred - target_emb;
        loss += diff * diff;
    }
    
    return loss / MODEL_DIM;
}

In this code, the model takes 64 bars of data, turns them into tokens, and adds a target — the price change in 24 bars. Tokens are converted into vectors via an embeddings matrix, then passed through six layers of Transformer. The error is calculated as the mean squared deviation between the forecast and the actual change. This helps the model learn, for example, by noticing that certain combinations of price changes in the past often lead to an increase 24 hours later.

Forecasting with the model

Once trained, the model is ready to make predictions. Here is how it works:

double Predict(const MqlRates &rates[], int current_pos) {
    if(!is_trained || current_pos < CONTEXT_LENGTH + FORECAST_HORIZON) {
        return 0.5; // Neutral prediction
    }
    
    // Prepare input tokens
    int tokens[CONTEXT_LENGTH];
    for(int i = 0; i < CONTEXT_LENGTH; i++) {
        int idx = current_pos - CONTEXT_LENGTH + i;
        if(idx > 0 && idx < ArraySize(rates)) {
            double change = (rates[idx].close - rates[idx-1].close) / rates[idx-1].close;
            tokens[i] = tokenizer.TokenizeChange(change);
        } else {
            tokens[i] = VOCAB_SIZE / 2; // Neutral token
        }
    }
    
    // Forward pass
    TGMatrix embedded;
    embedded.Init(CONTEXT_LENGTH, MODEL_DIM);
    
    // Position-encoded embeddings
    for(int pos = 0; pos < CONTEXT_LENGTH; pos++) {
        for(int dim = 0; dim < MODEL_DIM; dim++) {
            double emb = embeddings.Get(tokens[pos], dim);
            double pos_enc = MathSin(pos / MathPow(10000.0, 2.0 * dim / MODEL_DIM));
            embedded.Set(pos, dim, emb + 0.1 * pos_enc);
        }
    }
    
    // Pass through layers
    TGMatrix current;
    current.Copy(embedded);
    
    for(int i = 0; i < NUM_LAYERS; i++) {
        TGMatrix layer_out;
        layers[i].Forward(current, layer_out);
        current.Copy(layer_out);
    }
    
    // Get logits for the last position
    double logits[VOCAB_SIZE];
    for(int i = 0; i < VOCAB_SIZE; i++) {
        logits[i] = 0.0;
        for(int j = 0; j < MODEL_DIM; j++) {
            logits[i] += current.Get(CONTEXT_LENGTH - 1, j) * output_projection.Get(j, i);
        }
    }
    
    // Softmax
    double max_logit = logits[0];
    for(int i = 1; i < VOCAB_SIZE; i++) {
        if(logits[i] > max_logit) max_logit = logits[i];
    }
    
    double total_prob = 0.0;
    for(int i = 0; i < VOCAB_SIZE; i++) {
        logits[i] = MathExp(logits[i] - max_logit);
        total_prob += logits[i];
    }
    
    // Calculate the expected change
    double expected_change = 0.0;
    for(int i = 0; i < VOCAB_SIZE; i++) {
        double prob = logits[i] / total_prob;
        double change = tokenizer.DetokenizeChange(i);
        expected_change += prob * change;
    }
    
    // Convert to rise probability in 24 bars 
    double growth_prob = 0.5 + expected_change * 10.0; // Scale
    return MathMax(0.1, MathMin(0.9, growth_prob));
}

The model takes the last 64 bars, turns them into tokens, and adds positional encoding to know the order of the data. The data then passes through six layers of Transformer that analyze patterns. At output, the model produces probabilities for each token via the output_projection matrix. The Softmax function turns these numbers into probabilities, and the model calculates the expected price change and converts it into an increase probability (from 0.1 to 0.9). For example, a probability of 0.7 means that the model is 70% confident that the price will rise.

Optimization for MetaTrader 5

We chose a model dimension of MODEL_DIM = 256, six layers, and a context length of 64 bars to make the model fast and memory-efficient. Matrix operations in TGMatrix are optimized for sequential memory access, which speeds up computations. The Softmax function subtracts the maximum logit to avoid numerical issues. The DROPOUT_RATE = 0.1 parameter helps the model avoid overfitting by ignoring random data during training. These optimizations allow the model to train in 15 minutes and use approximately 200 MB of memory, making it accessible to typical computers.

Let's look at the test in MetaTrader 5:

Here are the backtest statistics:

A total of 48 trades were executed, 87% of which were profitable. The Sharpe ratio was 1.73 and the profit factor was 1.49.

TimeGPT efficiency and limitations

TimeGPT works because it combines the power of the Transformer architecture with adaptation for financial data. Tokenization simplifies complex prices, an attention mechanism finds important patterns, and Transformer layers make the model flexible. Historical data training allows it to adapt to the market, and optimizations make it practical for MetaTrader 5. The model achieves 68% accuracy in predicting price direction, which is better than many traditional methods.

However, to achieve maximum efficiency of the language model in financial markets, its dimension should be significantly larger than that of TimeGPT. Models with thousands or millions of parameters can capture even more complex patterns, but require powerful servers or specialized hardware, making them inaccessible to regular computers and laptops.

TimeGPT is optimized to run on limited resources, but this limits its ability to analyze very long-term trends or integrate additional data such as news or market sentiment. In the future, such models may become even more powerful, but this will require significant computing resources.

Table of included files:

File name	File usage
TimeGPT_EA.mql5	EA for trading the TimeGPT model. Place it in /Experts
TimeGPT.mqh	TimeGPT model. Place it in /Include