From Basic to Intermediate: Array (II)

Introduction

In the previous article From Basic to Intermediate: Array (I), we began discussing one of the most complex and challenging topics in programming. I know many might argue that it's actually quite simple and that it doesn't make sense for me to describe it as complicated. However, as we progress, you will understand why I assert that this topic is indeed complex and difficult to master. After all, it serves as the foundation for everything else.

Once I have explained and demonstrated how this concept can truly be applied, you, dear reader, who manage to fully understand what will be shown, will undoubtedly see why other features exist in a programming language. Because, once you understand this foundation, everything else becomes much easier to master and comprehend.

The greatest challenge here, in fact, is to present things in a way that doesn't delve into other topics that we have yet to cover. I am trying to illustrate why certain elements were created, without actually revealing them just yet. This is because understanding the underlying concept behind these tools is far more important than understanding the tools themselves. And as many programmers often ignore the concept and become overly focused on the tool, they end up at an impasse at times. This is because it's not the tool that solves the problem, but the concept. It's like a hammer - it can be used to drive nails. But it can also be used for other purposes - for example, to demolish things. Even though there might be a better tool for demolition, like a sledgehammer.

Before we begin, there is a prerequisite that is necessary to fully understand this article: it is essential to know and understand what a variable is and what a constant is.

ROM-Type Arrays

There are essentially two ways to declare an array. One is by declaring a static array, and the other is by declaring it as a dynamic array. While in practice, understanding each type is relatively straightforward, there are certain subtle nuances that can complicate or even prevent a clear understanding of what a dynamic array and a static array really are. Especially when considering other programming languages, such as C and C++. However, even here in MQL5, there may be moments where you find yourself somewhat unsure. This is because the fundamental and essential difference between a static and a dynamic array lies in the ability of the latter to change size during the execution of the code.

Thinking about it this way, it seems simple enough to classify an array as either dynamic or static. However, it's important to remember that a string is also an array. Though it's a special kind of array. This makes it complicated to classify it as strictly static or dynamic. Nonetheless, let's set this fact aside. We will not delve into the string type directly. Thus we will avoid confusion and to ensure we develop a clear understanding of this topic.

Fundamentally - and this is undisputed across any programming language you might study in the future - a constant array is always a static array. Regardless of the programming language you use, it will ALWAYS be static.

But why can we confidently assert that a constant array is always static? The reason is that a constant array, and you need to think of it this way, is essentially ROM memory. At this point, it might not make much sense. After all, our application is always loaded and executed in a region of RAM. So how can we imagine ROM memory, which only allows us to read data, within a RAM environment that lets us both read and write data at any point? This definitely seems illogical.

Precisely because of this, you must understand what constants and variables truly represent, dear reader. From the perspective of our application, it is indeed possible to use ROM-like behavior within RAM, without compromising the integrity or usability of the application. In fact, using constant arrays to create a small ROM within an application is relatively common. Particularly in more complex codebases or in applications that require highly specific values that must never change.

Think of the messages used to report events or information. You can consolidate these messages, translating them into different languages. When the application is loaded, it can determine which language to use and build the ROM memory accordingly. This would allow the application to be used by different users, each with a different language preference. This is a classic example of how an array might behave as though it were in ROM. However, if you instead use files to load the translations dynamically, the array would no longer be purely ROM-like. Or purely static. Although it could still be classified in this way.

To make this concept easier to understand, since understanding it is crucial for knowing how and when to use standard library calls to manage arrays, let's look at a simple example. It shows this concept in practice.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     const char Rom_01[]  = {72, 101, 108, 111, 33};
07.     const char Rom_02[8] = {'H', 'e', 'l', 'o', '!'};
08. 
09.     PrintFormat("%c%c%c%c%c%c", Rom_01[0], Rom_01[1], Rom_02[2], Rom_01[2], Rom_02[3], Rom_01[4]);
10. }
11. //+------------------------------------------------------------------+

Code 01

When Code 01 is executed, the result is shown just below.

Figure 01

This is quite easy to understand. However, we're interested here in lines six and seven. In both cases, we have two ROMs being created, each with the same apparent content. However, their sizes are completely different. But wait a minute. How can that be? I don't understand. Looking at the code, I can see that both arrays have five elements. And all the elements shown there are identical. Just they are declared in a different way. But why are these two ROMs different? It doesn't make sense.

You’re absolutely right, dear reader, that both arrays declare the same elements. Although in different ways. But what makes them different is precisely how they are declared. Now, pay close attention here to understand an important point.

ROM_01, which is declared on line six, contains five elements. Typically, in code where we create a static array, we see this kind of declaration. Notice that there's no value specified inside the brackets. But (and this is where many people get confused) not declaring a value inside the brackets for a constant array is perfectly normal. And even quite common. This is because it allows you to add more or fewer values to the array DURING THE IMPLEMENTATION. Once those values are set, they become completely locked in and cannot be modified. As a result, the array's size becomes fixed at the number of elements present.

Now, with ROM_02, things are a bit different. In this case, five elements are explicitly declared. However, since we’re specifying that the array has eight elements, three of these elements remain undefined. You need to be cautious when using these undefined elements. This is because they might contain valid data, or they might contain completely random values. This will depend on how the compiler initializes the array. Keep in mind that we are dealing with constant arrays here.

In any case, in both scenarios we are working with static arrays. That is, the number of elements WILL NOT CHANGE for the entire lifetime of the array. In one case, we will always have five elements, and in the other, always eight elements. Remember that the first element has an index of zero. So the counting always starts at zero.

Great. We know how many elements exist in each of the arrays. But what happens if we try to access the element at index five? Remember, counting starts from zero. Well, in this case, we can run a small test so you can observe what happens. To do this, let's change something in the code, so that it looks as shown below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     const char Rom_01[]  = {72, 101, 108, 111, 33};
07.     const char Rom_02[8] = {'H', 'e', 'l', 'o', '!'};
08. 
09.     const uchar pos = 6;
10. 
11.     PrintFormat("%c%c%c%c%c%c", Rom_01[0], Rom_01[1], Rom_02[2], Rom_01[2], Rom_02[3], Rom_01[4]);
12. 
13.     PrintFormat("Contents of position %d of the ROM_02 array: %d", pos, Rom_02[pos - 1]);
14.     PrintFormat("Contents of position %d of the ROM_01 array: %d", pos, Rom_01[pos - 1]);
15. }
16. //+------------------------------------------------------------------+

Code 02

Note that now, on line nine of Code 02, we have a constant indicating which element we are trying to access. In this case, we want the sixth element. It is at position five. You might think, "Well, since we're looking for the element at position five, and we have five elements declared, and we want to print the decimal value of this element, lines 13 and 14 will both print the same value - in this case, 33." Indeed, this is the most natural way to think. However, it's incorrect. This is because counting starts at zero. Therefore, the fifth declared element is actually the element with an index of four. So when we attempt to access index five, something will happen in the code. The result is shown below.

Figure 02

Note that here we have two strange things happening. The first is the fact that line 13 was executed, and the result shown is zero. This means the array declared on line seven contains hidden values. But the key point lies in the error message seen in this Figure 02. It tells us that on line 14 of Code 02, there is an attempt to access something outside the array.

Since the array being used on line 14 is the one declared on line six, you might be puzzled about why the error is occurring. This happens precisely because you are trying to access the element at the fifth position. Since the array has five elements, it might seem this should be possible. But once again, counting starts at zero.

And to confirm this, let's make a change. This change is in line 9 of code 02. By changing the value six, seen there, to the value five, as shown in the code line below, everything will change.

const uchar pos = 5;

When we compile and run Code 02 again, using the updated line shown above, the result in the terminal appears as follows.

Figure 03

Did you notice how now the code was able to correctly display the content? With this, I believe you'll be able to understand how we should access arrays. And how that small difference in array declaration can lead us to completely different outcomes.

Since ROM-type arrays are always static, even when declared in a way that might appear dynamic, there's not much more to say about them here. This is because this type of array, where any attempt to write to it results in a compilation error, leaves us with little else to discuss. So, let's move on to a new topic, where we will cover another type of array - one that is just a bit more complex. This is due to the fact that we can write to it.

RAM-Type Arrays

The prerequisite to understanding this type of array is, in fact, having understood ROM-type arrays. There the declaration can be either static or dynamic. However, unlike ROM-type arrays, whose content is constant and whose number of elements is also fixed, with RAM-type arrays things become a bit more complex. This is because the number of elements may or may not be constant. Also, the content of each element can vary as the code executes.

Now, with these introductory words, you might be thinking that this type of array is a living nightmare. However, it just requires a bit more attention to use correctly. This is where every detail makes all the difference. But there are certain specifics about arrays in MQL5 that I would like you to ignore for now, dear reader This is because there is a special type of array intended to function as a buffer. This is specific to MQL5. But this type will be better explained at another time, when we delve into calculations and programming designed for use in MetaTrader 5. So for now, what we'll explain here does not apply to these buffer-type arrays. The purpose here, at this moment, is to provide a general explanation of arrays in programming. Not just in MQL5, but across all programming languages.

Okay, so let's do the following: Based on the last code seen in the previous topic, let's remove the reserved word 'const' from both declarations. By doing this, we remove the limitations imposed on the arrays. However, there's a very important detail that needs to be explained when we make this change, which will be covered in the code shown next.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char Ram_01[]  = {72, 101, 108, 111, 33};
07.     char Ram_02[8] = {'H', 'e', 'l', 'o', '!'};
08. 
09.     const uchar pos = 5;
10. 
11.     PrintFormat("%c%c%c%c%c%c", Ram_01[0], Ram_01[1], Ram_02[2], Ram_01[2], Ram_02[3], Ram_01[4]);
12. 
13.     PrintFormat("Contents of position %d of the RAM_02 array: %d", pos, Ram_02[pos - 1]);
14.     PrintFormat("Contents of position %d of the RAM_01 array: %d", pos, Ram_01[pos - 1]);
15. }
16. //+------------------------------------------------------------------+

Code 03

When you run this code 03, you will get the same result as in Figure 03. But there is one fundamental difference. And it lies precisely in the fact that WE DO NOT WORK WITH ROM. Instead, we work with RAM. This allows us to change code 03 to create what is shown below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char Ram_01[]  = {72, 101, 108, 111, 33};
07.     char Ram_02[8] = {'H', 'e', 'l', 'o', '!'};
08. 
09.     uchar pos = 5;
10. 
11.     PrintFormat("%c%c%c%c%c%c", Ram_01[0], Ram_01[1], Ram_02[2], Ram_01[2], Ram_02[3], Ram_01[4]);
12. 
13.     PrintFormat("Contents of position %d of the Ram_02 array: %d", pos, Ram_02[pos - 1]);
14.     PrintFormat("Contents of position %d of the Ram_01 array: %d", pos, Ram_01[pos - 1]);
15. 
16.     Ram_02[pos - 1] = '$';
17.     PrintFormat("Contents of position %d of the Ram_02 array: %d", pos, Ram_02[pos - 1]);
18. 
19.     Ram_01[pos - 1] = '$';
20.     PrintFormat("Contents of position %d of the Ram_01 array: %d", pos, Ram_01[pos - 1]);
21. }
22. //+------------------------------------------------------------------+

Code 04

This Code 04 is quite curious, and in a way, even a bit intriguing. This is simply due to the fact that on lines 16 and 19, we are trying to modify the value at a given position in the array. This is a detail which, when well understood, can really spark the curiosity of the more daring among us. So, when we run Code 04, the result appears as shown below.

Figure 04

Notice that it was indeed possible to modify the values. This would not be allowed if we were using a ROM-type array. At this point, you might be wondering: Can we change any information within the array? Well, the answer to that question is: It depends.

Observe the following, dear reader: the array on line six, although it is dynamic, because we're not specifying an explicit element count, remains static. This is because we're initializing it at the time of declaration. The array on line seven is entirely static, but the number of possible elements within it is greater than the number of elements we are actually declaring during the array initialization.

Understanding this, you might say: So, if we use an index that points to an element, we can change its value. Right? Exactly. In fact, you could use code similar to what's shown below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char r[10] = {72, 101, 108, 108, 111, 33};
07. 
08.     string sz0;
09. 
10.     sz0 = "";
11.     for (uchar c = 0; c < ArraySize(r); c++)
12.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
13. 
14.     Print(sz0);
15. 
16.     Print("Modifying a value...");
17. 
18.     r[7] = 36;
19. 
20.     sz0 = "";
21.     for (uchar c = 0; c < ArraySize(r); c++)
22.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
23. 
24.     Print(sz0);
25. }
26. //+------------------------------------------------------------------+

Code 05

When we run Code 05, the result will be as shown in the image below.

Figure 05

In other words, it actually works. We can change any value, as long as the element we are accessing exists within the array. However, line 18 only worked because in line six, we specified that the array would have 10 elements.

Alright, I believe this first part is clear. But do we always need to work this way? That is, when declaring an array, do we always need to specify either the number of elements or the elements themselves? Indeed, this is a question that generates a lot of doubt among beginners And once again, the answer is: It depends.

However, and it's important that you remember this, dear reader: a constant array (that is, a ROM-type array) must ALWAYS have its elements or number of elements declared at the moment the array is created. One detail: declaring the elements themselves is mandatory in this case, but declaring the number of elements is optional.

This rule is strict and allows no changes or interpretations. However, regarding RAM-type arrays, there is no rigid rule to follow. Everything depends on the purpose or objective to be achieved. However, declaring elements in a dynamic array (an array that is not a ROM-type or constant) is not common. As we saw on line six of Code 04. This is because by doing so, we are essentially turning that dynamic array into a static one. This is somewhat similar to what happens with the array on line seven of the same Code 04.

However, in terms of code interpretation, an array declared as shown on line six of Code 04 could be replaced by something like what's shown below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char  Ram_01[5],
07.           pos = 5;
08.     
09.     Ram_01[0] = 72;
10.     Ram_01[1] = 101;
11.     Ram_01[2] = 108;
12.     Ram_01[3] = 111;
13.     Ram_01[4] = 33;
14. 
15.     PrintFormat("%c%c%c%c%c%c", Ram_01[0], Ram_01[1], Ram_01[2], Ram_01[2], Ram_01[3], Ram_01[4]);
16. 
17.     PrintFormat("Contents of position %d of the Ram_01 array: %d", pos, Ram_01[pos - 1]);
18.     Ram_01[pos - 1] = '$';
19.     PrintFormat("Contents of position %d of the Ram_01 array: %d", pos, Ram_01[pos - 1]);
20. }
21. //+------------------------------------------------------------------+

Code 06

Note that in Code 06, we now have a static array declaration on line six. By doing this, we are instructing the compiler to allocate a memory block for us. This allocation happens automatically during compilation. Thus, the compiler will reserve a space in memory large enough to contain the specified number of elements. In this case, five elements. Since the type is 'char', we will have five bytes allocated and available for immediate use.

Now think of this allocated space as a five-byte variable. You can use it however it suits your needs. We could also allocate an arbitrary number. For example, if instead of five, as seen on line six, we used 100, we would be creating what could be considered a 100-byte variable. Keep in mind that the current largest width is 8 bytes, or 64 bits. In other words, we could fit just over 12 of these 8-byte variables inside this 100-byte array.

In any case, when you run Code 06, you will see the result shown below.

Figure 06

Now note that the initialization was not done during the array declaration, but rather between lines nine and thirteen. This works the same way as line 18 in Code 06, and just as it was done in the previous examples.

However, there is one small detail here Since the array is being created in a static manner, if you need more space, for any reason, you will not be able to allocate additional memory. This limitation appears at runtime. For this reason, static arrays are used in very specific situations. The same applies to dynamic arrays. But what if, instead of defining a value to indicate the number of elements in the array, as seen on line six of Code 06, we used the line shown below?

    char  Ram_01[]

This would result in the declaration of a fully dynamic array. Now, pay close attention, dear reader. By doing this, we are telling the compiler that we, the programmers, will be responsible for managing the array. i.e. allocating and freeing memory as needed. However, as soon as line nine is executed, an error will occur. This is because you will be attempting to access memory that has not yet been allocated. This error can be seen below.

Figure 07

It's not uncommon, especially in programs with many variables, to encounter this type of error. However, fixing it is simple. All we need to do is allocate the memory before using the array. And then, once we are finished with the array, we should explicitly free the memory. This explicit deallocation is considered a good programming practice.

In simpler programs, or in less professional code, it's common to see this deallocation step skipped. However, this is not recommended, since memory allocated will not be released automatically just because you've stopped using that region of memory. So, it's best to develop good habits from the start. Allocate memory when needed, and free it when it's no longer required, to avoid strange errors during code execution.

Finally, let's look at how to fix the code to use the dynamic array correctly. We replace code 06 with code 07.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char   r[];
07.     string sz0;
08.     
09.     PrintFormat("Allocated enough position for %d elements", ArrayResize(r, 10));
10. 
11.     r[0] = 72;
12.     r[1] = 101;
13.     r[2] = 108;
14.     r[3] = 111;
15.     r[4] = 33;
16. 
17.     sz0 = "";
18.     for (uchar c = 0; c < ArraySize(r); c++)
19.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
20.     Print(sz0);
21. 
22.     Print("Modifying a value...");
23. 
24.     r[7] = 36;
25. 
26.     sz0 = "";
27.     for (uchar c = 0; c < ArraySize(r); c++)
28.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
29.     Print(sz0);
30. 
31.     ArrayFree(r);
32. }
33. //+------------------------------------------------------------------+

Code 07

This code snippet, my dear reader, truly makes use of a purely dynamic array. Notice how it closely resembles Code 05. This is done intentionally. This similarity is intentional, designed to demonstrate that we can achieve the same outcome in different ways. The result of executing Code 07 is shown below.

Figure 08

But pay close attention to the highlighted information in Figure 08. This part is extremely important. Normally, less cautious programmers, or those writing code not intended for critical purposes, will use memory without properly initializing it. This kind of oversight leads to an absurd number of hard-to-detect bugs, even for highly experienced developers. Here, I am demonstrating one such issue.

Notice that memory was allocated on line 9 of Code 07. Between lines 11 and 15, we assign values to some positions in the allocated memory. However, when we read from the memory, trying to retrieve its contents, we see something strange there. This information is called garbage because it exists in the memory, but it shouldn't be there.

The key point here is line 22. It tells us that, at that moment, we will be modifying the memory. This occurs on line 24. However, upon reading the memory again, it's as if the program could predict the future. As if, before we applied the value to that memory location, the value appeared from nowhere. As if by magic.

So, why did this happen? It's not a magic trick, nor is it time travel. The explanation lies in the fact that the allocated memory is not truly under your control. The operating system handles the memory allocation, and the contents of that memory can be anything. This can be leftover data from other programs (known as garbage) or even a bunch of zeros. The contents are always random. Moreover, if the operating system allocates the same memory location for two consecutive executions, where the memory was allocated, freed, and then allocated again, whether by the same code or a completely different one, the chance of having garbage is very high.

If you assume the contents of that memory are a certain way, you may encounter a very difficult-to-resolve problem. Therefore, NEVER assume memory contains a specific value. NEVER. Always, when allocating memory, CLEAN the region or at least initialize it completely. There are many ways to do this in MQL5. Personally, I prefer a simple and efficient call from the standard library to handle this. So, the corrected version of Code 07 is shown below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. //+------------------------------------------------------------------+
04. void OnStart(void)
05. {
06.     char   r[];
07.     string sz0;
08.     
09.     PrintFormat("Allocated enough position for %d elements", ArrayResize(r, 10));
10. 
11.     ZeroMemory(r);
12. 
13.     r[0] = 72;
14.     r[1] = 101;
15.     r[2] = 108;
16.     r[3] = 111;
17.     r[4] = 33;
18. 
19.     sz0 = "";
20.     for (uchar c = 0; c < ArraySize(r); c++)
21.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
22.     Print(sz0);
23. 
24.     Print("Modifying a value...");
25. 
26.     r[7] = 36;
27. 
28.     sz0 = "";
29.     for (uchar c = 0; c < ArraySize(r); c++)
30.         sz0 = StringFormat("%s%03d, ", sz0, r[c]);
31.     Print(sz0);
32. 
33.     ArrayFree(r);
34. }
35. //+------------------------------------------------------------------+

Code 08

When you run code 08, you will get what you see below.

Figure 09

Notice that the only difference between Image 08 and Image 09 is the specific memory location. In the first run, we encountered garbage data in that location. However, after adding line 11 in Code 08, this kind of error, where garbage could be used, no longer occurs. It's that simple. That said, there are other functions in MQL5 that serve the same purpose. It all depends on your choice and the goal you want to achieve.

Final Considerations

In this article, which indeed touches on the foundation of something much more complex than what has been shown here, I demonstrated how you can approach arrays from a more professional perspective. Although you might not have fully understood everything I intended to show, as this kind of topic is far more challenging to explain than many other programming concepts. However, I believe I've achieved part of the main objective. I explained how to create ROM-like behavior within RAM. We also discussed the decision-making process for choosing between dynamic and static arrays.

That said, it's important to highlight one key point: never assume anything when dealing with arrays or memory access. Uninitialized elements can contain garbage. And using such garbage in your code can seriously affect your expected outcomes or make it much harder to resolve interactions between different programs.

In any case, the attached materials include the codes we reviewed here. This will allow you to study and practice. This will also help you learn how to handle failures and to explore concepts related to array usage in more practical tasks.