Neuron Net
Public Member Functions | Protected Member Functions | Protected Attributes | List of all members
CNeuronAttentionOCL Class Reference

Class of Self-Attention layer GPU calculation. More...

Inheritance diagram for CNeuronAttentionOCL:
CNeuronBaseOCL

Public Member Functions

 CNeuronAttentionOCL (void)
 Constructor. More...
 
 ~CNeuronAttentionOCL (void)
 Destructor. More...
 
virtual bool Init (uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window, uint units_count, ENUM_OPTIMIZATION optimization_type)
 Method of initialization class. More...
 
virtual bool calcInputGradients (CNeuronBaseOCL *prevLayer)
 Method to transfer gradients to previous layer. More...
 
virtual int Type (void) const
 Identificator of class. More...
 
virtual bool Save (int const file_handle)
 Save method. More...
 
virtual bool Load (int const file_handle)
 Load method. More...
 
virtual bool Init (uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint numNeurons, ENUM_OPTIMIZATION optimization_type)
 Method of initialization class. More...
 
virtual void SetActivationFunction (ENUM_ACTIVATION value)
 Set the type of activation function (ENUM_ACTIVATION) More...
 
virtual int getOutputIndex (void)
 Get index of output buffer. More...
 
virtual int getPrevOutIndex (void)
 Get index of previous iteration output buffer. More...
 
virtual int getGradientIndex (void)
 Get index of gradient buffer. More...
 
virtual int getWeightsIndex (void)
 Get index of weights matrix buffer. More...
 
virtual int getDeltaWeightsIndex (void)
 Get index of delta weights matrix buffer (SGD) More...
 
virtual int getFirstMomentumIndex (void)
 Get index of first momentum matrix buffer (Adam) More...
 
virtual int getSecondMomentumIndex (void)
 Get index of Second momentum matrix buffer (Adam) More...
 
virtual int getOutputVal (double &values[])
 Get values of output buffer. More...
 
virtual int getOutputVal (CArrayDouble *values)
 Get values of output buffer. More...
 
virtual int getPrevVal (double &values[])
 Get values of previous iteration output buffer. More...
 
virtual int getGradient (double &values[])
 Get values of gradient buffer. More...
 
virtual int getWeights (double &values[])
 Get values of weights matrix buffer. More...
 
virtual int Neurons (void)
 Get number of neurons in layer. More...
 
virtual int Activation (void)
 Get type of activation function. More...
 
virtual int getConnections (void)
 Get number of connections 1 neuron to next layer. More...
 
virtual bool FeedForward (CObject *SourceObject)
 Dispatch method for defining the subroutine for feed forward process. More...
 
virtual bool calcHiddenGradients (CObject *TargetObject)
 Dispatch method for defining the subroutine for transferring the gradient to the previous layer. More...
 
virtual bool calcHiddenGradients (CNeuronBaseOCL *NeuronOCL)
 Method to transfer gradient to previous layer by calling kernel CalcHiddenGradient(). More...
 
virtual bool UpdateInputWeights (CObject *SourceObject)
 Dispatch method for defining the subroutine for updating weights. More...
 
virtual bool calcOutputGradients (CArrayDouble *Target)
 Method of output gradients calculation by calling kernel CalcOutputGradient(). More...
 

Protected Member Functions

virtual bool feedForward (CNeuronBaseOCL *prevLayer)
 Feed Forward method. More...
 
virtual bool updateInputWeights (CNeuronBaseOCL *prevLayer)
 Method for updating weights. More...
 

Protected Attributes

CNeuronConvOCLQuerys
 Convolution layer for Querys. More...
 
CNeuronConvOCLKeys
 Convolution layer for Keys. More...
 
CNeuronConvOCLValues
 Convolution layer for Values. More...
 
CBufferDoubleScores
 Buffer for Scores matrix. More...
 
CNeuronBaseOCLAttentionOut
 Layer of Self-Attention Out. More...
 
CNeuronConvOCLFF1
 Convolution layer for first layer of Feed Forward block. More...
 
CNeuronConvOCLFF2
 Convolution layer for second layer of Feed Forward block. More...
 
uint iWindow
 Window size. More...
 
uint iUnits
 Number of units. More...
 
COpenCLMyOpenCL
 Object for working with OpenCL. More...
 
CBufferDoubleOutput
 Buffer of Output tenzor. More...
 
CBufferDoublePrevOutput
 Buffer of previous iteration Output tenzor. More...
 
CBufferDoubleWeights
 Buffer of weights matrix. More...
 
CBufferDoubleDeltaWeights
 Buffer of last delta weights matrix (SGD) More...
 
CBufferDoubleGradient
 Buffer of gradient tenzor. More...
 
CBufferDoubleFirstMomentum
 Buffer of first momentum matrix (ADAM) More...
 
CBufferDoubleSecondMomentum
 Buffer of second momentum matrix (ADAM) More...
 
const double alpha
 Multiplier to momentum in SGD optimization. More...
 
int t
 Count of iterations. More...
 
int m_myIndex
 Index of neuron in layer. More...
 
ENUM_ACTIVATION activation
 Activation type (ENUM_ACTIVATION) More...
 
ENUM_OPTIMIZATION optimization
 Optimization method (ENUM_OPTIMIZATION) More...
 

Detailed Description

Class of Self-Attention layer GPU calculation.

Detailed description on the link.

Definition at line 3890 of file NeuroNet.mqh.

Constructor & Destructor Documentation

◆ CNeuronAttentionOCL()

CNeuronAttentionOCL::CNeuronAttentionOCL ( void  )
inline

Constructor.

Definition at line 3908 of file NeuroNet.mqh.

◆ ~CNeuronAttentionOCL()

CNeuronAttentionOCL::~CNeuronAttentionOCL ( void  )

Destructor.

Definition at line 3922 of file NeuroNet.mqh.

Member Function Documentation

◆ Activation()

virtual int CNeuronBaseOCL::Activation ( void  )
inlinevirtualinherited

Get type of activation function.

Returns
Type (ENUM_ACTIVATION)

Definition at line 2973 of file NeuroNet.mqh.

◆ calcHiddenGradients()

bool CNeuronBaseOCL::calcHiddenGradients ( CObject *  TargetObject)
virtualinherited

Dispatch method for defining the subroutine for transferring the gradient to the previous layer.

Parameters
TargetObjectPointer to the next layer.

Definition at line 3328 of file NeuroNet.mqh.

◆ calcInputGradients()

bool CNeuronAttentionOCL::calcInputGradients ( CNeuronBaseOCL prevLayer)
virtual

Method to transfer gradients to previous layer.

Parameters
[in]prevLayerPointer to previous layer.

Definition at line 4136 of file NeuroNet.mqh.

◆ feedForward()

bool CNeuronAttentionOCL::feedForward ( CNeuronBaseOCL prevLayer)
protectedvirtual

Feed Forward method.

Parameters
prevLayerPointer to previos layer.

Reimplemented from CNeuronBaseOCL.

Definition at line 4028 of file NeuroNet.mqh.

◆ FeedForward()

bool CNeuronBaseOCL::FeedForward ( CObject *  SourceObject)
virtualinherited

Dispatch method for defining the subroutine for feed forward process.

Parameters
SourceObjectPointer to the previous layer.

Definition at line 3151 of file NeuroNet.mqh.

◆ getConnections()

virtual int CNeuronBaseOCL::getConnections ( void  )
inlinevirtualinherited

Get number of connections 1 neuron to next layer.

Returns
Number of connections

Definition at line 2974 of file NeuroNet.mqh.

◆ getDeltaWeightsIndex()

virtual int CNeuronBaseOCL::getDeltaWeightsIndex ( void  )
inlinevirtualinherited

Get index of delta weights matrix buffer (SGD)

Returns
Index

Definition at line 2963 of file NeuroNet.mqh.

◆ getFirstMomentumIndex()

virtual int CNeuronBaseOCL::getFirstMomentumIndex ( void  )
inlinevirtualinherited

Get index of first momentum matrix buffer (Adam)

Returns
Index

Definition at line 2964 of file NeuroNet.mqh.

◆ getGradient()

virtual int CNeuronBaseOCL::getGradient ( double &  values[])
inlinevirtualinherited

Get values of gradient buffer.

Parameters
[out]valuesArray of data
Returns
number of items

Definition at line 2970 of file NeuroNet.mqh.

◆ getGradientIndex()

virtual int CNeuronBaseOCL::getGradientIndex ( void  )
inlinevirtualinherited

Get index of gradient buffer.

Returns
Index

Definition at line 2961 of file NeuroNet.mqh.

◆ getOutputIndex()

virtual int CNeuronBaseOCL::getOutputIndex ( void  )
inlinevirtualinherited

Get index of output buffer.

Returns
Index

Definition at line 2959 of file NeuroNet.mqh.

◆ getOutputVal() [1/2]

virtual int CNeuronBaseOCL::getOutputVal ( CArrayDouble *  values)
inlinevirtualinherited

Get values of output buffer.

Parameters
[out]valuesArray of data
Returns
number of items

Definition at line 2968 of file NeuroNet.mqh.

◆ getOutputVal() [2/2]

virtual int CNeuronBaseOCL::getOutputVal ( double &  values[])
inlinevirtualinherited

Get values of output buffer.

Parameters
[out]valuesArray of data
Returns
number of items

Definition at line 2967 of file NeuroNet.mqh.

◆ getPrevOutIndex()

virtual int CNeuronBaseOCL::getPrevOutIndex ( void  )
inlinevirtualinherited

Get index of previous iteration output buffer.

Returns
Index

Definition at line 2960 of file NeuroNet.mqh.

◆ getPrevVal()

virtual int CNeuronBaseOCL::getPrevVal ( double &  values[])
inlinevirtualinherited

Get values of previous iteration output buffer.

Parameters
[out]valuesArray of data
Returns
number of items

Definition at line 2969 of file NeuroNet.mqh.

◆ getSecondMomentumIndex()

virtual int CNeuronBaseOCL::getSecondMomentumIndex ( void  )
inlinevirtualinherited

Get index of Second momentum matrix buffer (Adam)

Returns
Index

Definition at line 2965 of file NeuroNet.mqh.

◆ getWeights()

virtual int CNeuronBaseOCL::getWeights ( double &  values[])
inlinevirtualinherited

Get values of weights matrix buffer.

Parameters
[out]valuesArray of data
Returns
number of items

Definition at line 2971 of file NeuroNet.mqh.

◆ getWeightsIndex()

virtual int CNeuronBaseOCL::getWeightsIndex ( void  )
inlinevirtualinherited

Get index of weights matrix buffer.

Returns
Index

Definition at line 2962 of file NeuroNet.mqh.

◆ Init() [1/2]

bool CNeuronBaseOCL::Init ( uint  numOutputs,
uint  myIndex,
COpenCLMy open_cl,
uint  numNeurons,
ENUM_OPTIMIZATION  optimization_type 
)
virtualinherited

Method of initialization class.

Parameters
[in]numOutputsNumber of connections to next layer.
[in]myIndexIndex of neuron in layer.
[in]open_clPointer to COpenCLMy object. #param[in] numNeurons Number of neurons in layer
optimization_typeOptimization type (ENUM_OPTIMIZATION)
Returns
Boolen result of operations.

Definition at line 3031 of file NeuroNet.mqh.

◆ Init() [2/2]

bool CNeuronAttentionOCL::Init ( uint  numOutputs,
uint  myIndex,
COpenCLMy open_cl,
uint  window,
uint  units_count,
ENUM_OPTIMIZATION  optimization_type 
)
virtual

Method of initialization class.

Parameters
[in]numOutputsNumber of connections to next layer.
[in]myIndexIndex of neuron in layer.
[in]open_clPointer to COpenCLMy object.
[in]windowSize of in/out window and step.
[in]units_countNumberof neurons.
[in]optimization_typeOptimization type (ENUM_OPTIMIZATION)
Returns
Boolen result of operations.

Definition at line 3942 of file NeuroNet.mqh.

◆ Load()

bool CNeuronAttentionOCL::Load ( int const  file_handle)
virtual

Load method.

Parameters
[in]file_handlehandle of file
Returns
logical result of operation

Reimplemented from CNeuronBaseOCL.

Definition at line 4336 of file NeuroNet.mqh.

◆ Neurons()

virtual int CNeuronBaseOCL::Neurons ( void  )
inlinevirtualinherited

Get number of neurons in layer.

Returns
Number of neurons

Definition at line 2972 of file NeuroNet.mqh.

◆ Save()

bool CNeuronAttentionOCL::Save ( int const  file_handle)
virtual

Save method.

Parameters
[in]file_handlehandle of file
Returns
logical result of operation

Reimplemented from CNeuronBaseOCL.

Definition at line 4308 of file NeuroNet.mqh.

◆ SetActivationFunction()

virtual void CNeuronBaseOCL::SetActivationFunction ( ENUM_ACTIVATION  value)
inlinevirtualinherited

Set the type of activation function (ENUM_ACTIVATION)

Definition at line 2957 of file NeuroNet.mqh.

◆ Type()

virtual int CNeuronAttentionOCL::Type ( void  ) const
inlinevirtual

Identificator of class.

Returns
Type of class

Reimplemented from CNeuronBaseOCL.

Definition at line 3914 of file NeuroNet.mqh.

◆ updateInputWeights()

bool CNeuronAttentionOCL::updateInputWeights ( CNeuronBaseOCL prevLayer)
protectedvirtual

Method for updating weights.

Parameters
prevLayerPointer to previos layer.

Reimplemented from CNeuronBaseOCL.

Definition at line 4275 of file NeuroNet.mqh.

Member Data Documentation

◆ activation

ENUM_ACTIVATION CNeuronBaseOCL::activation
protectedinherited

Activation type (ENUM_ACTIVATION)

Definition at line 2943 of file NeuroNet.mqh.

◆ alpha

const double CNeuronBaseOCL::alpha
protectedinherited

Multiplier to momentum in SGD optimization.

Definition at line 2939 of file NeuroNet.mqh.

◆ AttentionOut

CNeuronBaseOCL* CNeuronAttentionOCL::AttentionOut
protected

Layer of Self-Attention Out.

Definition at line 3897 of file NeuroNet.mqh.

◆ DeltaWeights

CBufferDouble* CNeuronBaseOCL::DeltaWeights
protectedinherited

Buffer of last delta weights matrix (SGD)

Definition at line 2933 of file NeuroNet.mqh.

◆ FF1

CNeuronConvOCL* CNeuronAttentionOCL::FF1
protected

Convolution layer for first layer of Feed Forward block.

Definition at line 3898 of file NeuroNet.mqh.

◆ FF2

CNeuronConvOCL* CNeuronAttentionOCL::FF2
protected

Convolution layer for second layer of Feed Forward block.

Definition at line 3899 of file NeuroNet.mqh.

◆ FirstMomentum

CBufferDouble* CNeuronBaseOCL::FirstMomentum
protectedinherited

Buffer of first momentum matrix (ADAM)

Definition at line 2935 of file NeuroNet.mqh.

◆ Gradient

CBufferDouble* CNeuronBaseOCL::Gradient
protectedinherited

Buffer of gradient tenzor.

Definition at line 2934 of file NeuroNet.mqh.

◆ iUnits

uint CNeuronAttentionOCL::iUnits
protected

Number of units.

Definition at line 3902 of file NeuroNet.mqh.

◆ iWindow

uint CNeuronAttentionOCL::iWindow
protected

Window size.

Definition at line 3901 of file NeuroNet.mqh.

◆ Keys

CNeuronConvOCL* CNeuronAttentionOCL::Keys
protected

Convolution layer for Keys.

Definition at line 3894 of file NeuroNet.mqh.

◆ m_myIndex

int CNeuronBaseOCL::m_myIndex
protectedinherited

Index of neuron in layer.

Definition at line 2942 of file NeuroNet.mqh.

◆ OpenCL

COpenCLMy* CNeuronBaseOCL::OpenCL
protectedinherited

Object for working with OpenCL.

Definition at line 2929 of file NeuroNet.mqh.

◆ optimization

ENUM_OPTIMIZATION CNeuronBaseOCL::optimization
protectedinherited

Optimization method (ENUM_OPTIMIZATION)

Definition at line 2944 of file NeuroNet.mqh.

◆ Output

CBufferDouble* CNeuronBaseOCL::Output
protectedinherited

Buffer of Output tenzor.

Definition at line 2930 of file NeuroNet.mqh.

◆ PrevOutput

CBufferDouble* CNeuronBaseOCL::PrevOutput
protectedinherited

Buffer of previous iteration Output tenzor.

Definition at line 2931 of file NeuroNet.mqh.

◆ Querys

CNeuronConvOCL* CNeuronAttentionOCL::Querys
protected

Convolution layer for Querys.

Definition at line 3893 of file NeuroNet.mqh.

◆ Scores

CBufferDouble* CNeuronAttentionOCL::Scores
protected

Buffer for Scores matrix.

Definition at line 3896 of file NeuroNet.mqh.

◆ SecondMomentum

CBufferDouble* CNeuronBaseOCL::SecondMomentum
protectedinherited

Buffer of second momentum matrix (ADAM)

Definition at line 2936 of file NeuroNet.mqh.

◆ t

int CNeuronBaseOCL::t
protectedinherited

Count of iterations.

Definition at line 2940 of file NeuroNet.mqh.

◆ Values

CNeuronConvOCL* CNeuronAttentionOCL::Values
protected

Convolution layer for Values.

Definition at line 3895 of file NeuroNet.mqh.

◆ Weights

CBufferDouble* CNeuronBaseOCL::Weights
protectedinherited

Buffer of weights matrix.

Definition at line 2932 of file NeuroNet.mqh.


The documentation for this class was generated from the following file: