úÎîI嫉            ! " # $ % & ' ( )*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…† ‡ ˆ ! Common Types(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone eMatrix Types (of Doubles)Vector Types (of Doubles)Scalar Type (Double)Regularization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone³RegularizationNo regularizationL2 Regularization`Calculates regularization for Model.cost function. It takes regularization parameter and theta.dCalculates regularization for Model.gradient function. It takes regularization parameter and theta.$Random generation utility functions.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoney Samples n+ (given as a second parameter) values from list (given as a third parameter). Samples n+ (given as a second parameter) values from list1 (given as a third parameter) inside RandomMonad. AReturns a list of random values distributed in a closed interval range CReturns a vector of random values distributed in a closed interval range CReturns a matrix of random values distributed in a closed interval range‰…Takes a list of ranges `(lo, hi)`, returns a list of random values uniformly distributed in the list of closed intervals [(lo, hi)]. list's lengthsrange(list of random values inside RandomMonad vector's lengthrange number of rowsnumber of columnsrange  Weight Initialization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone%1<Weight Initialization Algorithm discussed in Nguyen at al.:  :https://web.stanford.edu/class/ee373b/nninitialization.pdf¦ Nguyen, Derrick, Widrow, B. Improving the learning speed of 2-layer neural networks by choosing initial values of adaptive weights. In Proc. IJCNN, 1990; 3: 21-26.8Weight Initialization Algorithm discussed in He at al.:  https://arxiv.org/abs/1502.01852’ Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.Tanh Activation.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone&‡ReLu Activation(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone'ÙRegularization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone,sfCalcaulates regularization for forward propagation. It takes regularization parameter and theta list.iCalculates regularization for step of backward propagation. It takes regularization parameter and theta.Regression Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone4-{Hypothesis function, a.k.a. score function (for lassifition problem) Takes X (m x n) and theta (n x 1), returns y (m x 1).ŽCost function J(Theta), a.k.a. loss function. It takes regularizarion parameter, matrix X (m x n), vector y (m x 1) and vector theta (n x 1).„Gradient function. It takes regularizarion parameter, X (m x n), y (m x 1) and theta (n x 1). Returns vector of gradients (n x 1). Gradient Descent(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone@Ž6Minibatch Gradient Descent method implementation. See MachineLearning.Regression for usage details.Š6Minibatch Gradient Descent method implementation. See MachineLearning.Regression for usage details. seed batch sizelearning rate, alphamodel to learnepsilonmax number of iters regularization parameter, lambdamatrix of features, Xoutput vector, y %vector of initial weights, theta or w $vector of weights and learning path Š  batch sizelearning rate, alphamodel to learnepsilonmax number of iters regularization parameter, lambdamatrix of features, Xoutput vector, y%vector of initial weights, theta or w $vector of weights and learning path  Gradient Descent(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneCN,Gradient Descent method implementation. See MachineLearning.Regression for usage details.  Optimization(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneV¡>Gradient descent, takes alpha. Requires feature normalization.<Minibacth Gradietn Descent, takes seed, batch size and alpharFletcher-Reeves conjugate gradient algorithm, takes size of first trial step (0.1 is fine) and tol (0.1 is fine).pPolak-Ribiere conjugate gradient algorithm. takes size of first trial step (0.1 is fine) and tol (0.1 is fine). wBroyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, takes size of first trial step (0.1 is fine) and tol (0.1 is fine).!ŒReturns solution vector (theta) and optimization path. Optimization path's row format: [iter number, cost function value, theta values...]"ÿ,Gradient checking function. Approximates the derivates of the Model's cost function and calculates derivatives using the Model's gradient functions. Returns norn_2 between 2 derivatives. Takes model, regularization, X, y, theta and epsilon (used to approximate derivatives, 1e-4 is a good value).!.model (Least Squares, Logistic Regression etc)*epsilon, desired precision of the solution$maximum number of iterations allowedregularization parameterXyinitial solution, theta%solution vector and optimization path !" !" Logistic Regression Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneY@%Calculates sigmoid&!Calculates derivatives of sigmoid#$%&#$%&#$ Sigmoid(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneZ›%(%(Logistic Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone[é)*+)*+Least Squares Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone]S,-,-,- Regression"(c) Alexander Ignatyev, 2016-2018.BSD-3 experimentalPOSIXNoneaö/gNormal equation using inverse, does not require feature normalization It takes X and y, returns theta.0fNormal equation using pseudo inverse, requires feature normalization It takes X and y, returns theta. !,-/0,- !/0 Clustering(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNonew¸ 1;Cluster type (list of samples associtaed with the cluster).‹0Gets list if the nearest centroid to the sample.Œ,Calculates cost associated with one claster.'Calculates cost value for all clusters.Ž8Calculates centroid (middle point) of the given cluster.2Calculates new cluster centroids for each cluster.1Build cluster list from list of clusters indices.2/Run K-Means algorithm once inside Random Monad.3SClusters data using K-Means Algorithm inside Random Monad. Runs K-Means algorithm N1 times, returns the clustering with smaller cost. ‹list of cluster cetroids;sample;index of the nearest centroid.Œcluster;cluster centroid; cost value. cluster list;list of cluster centroids; cost value.Žcluster; centroid.list of clusters;list of cluster centroids.list of samples;Clist of cluster indices (associated cluster index for each sample);list of clusters.‘list of samples;number of clusters (K);list of initial centroids; (list of clusters, cost values).2list of samples;number of clusters (K);iteration number;4(list of clusters, cost values) inside Random Monad.3"number of K-Means Algorithm runs (N);data to cluster;desired number of clusters (K);%list of clusters inside Random Monad.123132!Classification Internal module.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNone€r4åCalculates accuracy of Classification predictions. Takes vector expected y and vector predicted y. Returns number from 0 to 1, the closer to 1 the better accuracy. Suitable for both Classification Types: Binary and Multiclass.’ÚProcess outputs for One-vs-All Classification. Takes number of labels and output vector y. Returns list of vectors of binary outputs (One-vs-All Classification). It is supposed that labels are integerets start at 0.4’.Learn function with progress bar for terminal.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone‹‰5¼Learn the given function displaying progress bar in terminal. It takes function, initial theta and number of iterations to call the function. It returns theta and optimization path (see MachineLearning.Optimization for details).6ëLearn the given function displaying progress bar in terminal. It takes function, list of outputs and list of initial thetas and number of iterations to call the function. It returns list of thetas and list of optimization paths (see MachineLearning.Optimization for details).“PBuild a single optimazation path matrix from list of optimization path matrices.5656One-vs-All Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNone”X7±One-vs-All Classification prediction function. Takes a matrix of features X and a list of vectors theta, returns predicted class number assuming that class numbers start at 0.8 Learns One-vs-All Classification8 (e.g. BFGS2 0.1 0.1)+epsilon, desired precision of the solution;%maximum number of iterations allowed; regularization parameter lambda;number of labels matrix X;vector yinitial theta list;&solution vector and optimization path.  478  784Binary Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNoneœ/9vBinary Classification prediction function. Takes a matrix of features X and a vector theta. Returns predicted class.:Learns Binary Classification.:(e.g. BFGS2 0.1 0.1)+epsilon, desired precision of the solution;%maximum number of iterations allowed; regularization parameter lambda; matrix X;binary vector y;initial Theta;&solution vector and optimization path.  49:  9:4Machine Learning(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone¢ù;'Add biad dimension to the future matrix<Remove biad dimension=zCaclulates mean and stddev values of every feature. Takes feature matrix X, returns pair of vectors of means and stddevs.?JMaps the features into all polynomial terms of X up to the degree-th power@?Splits data matrix to features matrix X and vector of outputs y;<=>?@;<=>?@Principal Component Analysis."(c) Alexander Ignatyev, 2017-2018.BSD-3 experimentalPOSIXNoneª÷”Computes "covariance matrix".•^Compute eigenvectors (matrix U) and singular values (matrix S) of the given covariance matrix.A†Gets dimensionality reduction function, retained variance (0..1) and reduced X for given matrix X and number of dimensions to retain.B†Gets dimensionality reduction function, retained number of dimensions and reduced X for given matrix X and variance to retain (0..1].ABABMultiClass Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNoneº½ C8MultiClassModel is Model wrapper class around ClassifierECClassifier type class represents Multi-class classification models.FScore functionGJHypothesis function Takes X (m x n) and theta (n x k), returns y (m x k).H•Cost function J(Theta), a.k.a. loss function. It takes regularizarion parameter lambda, matrix X (m x n), vector y (m x 1) and vector theta (n x 1).I‹Gradient function. It takes regularizarion parameter lambda, X (m x n), y (m x 1) and theta (n x 1). Returns vector of gradients (n x 1).JReturns Number of ClassesK¯Process outputs for MultiClass Classification. Takes Classifier and output vector y. Returns matrix of binary outputs. It is supposed that labels are integerets start at 0.L]Calculates regularization for Classifier.ccost. It takes regularization parameter and theta.MaCalculates regularization for Classifier.cgradient. It takes regularization parameter and theta.CDEFGHIJKLMEFGHIJCDKLMCDEFGHIJUtils(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone¼ùU"Converts list of tuples into list.OPQRSTUOPQRTSUSoftmax Classifier."(c) Alexander Ignatyev, 2017-2018.BSD-3 experimentalPOSIXNone¿SV,Softmax Classifier, takes number of classes.CDEFGHIJKLMVWVWVW Softmax Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneÀîYZ[YZ[Multi SVM Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneÂt– SVM Delta\]^\]^Neural Network's Layer"(c) Alexander Ignatyev, 2017-2018.BSD-3 experimentalPOSIXNoneQVÄ_`abcdefghijklm_`abcdefghijklm_`abcdefghijkNeural Network's Topology(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneÚªn‡Neural network topology has at least 2 elements: numver of input and number of outputs. And sizes of hidden layers between 2 elements.o.Loss function's type. Takes x, weights and y.ppMakes Neural Network's Topology. Takes number of inputs, list of hidden layers, output layer and loss function.qWCalculates loss for the given topology. Takes topology, regularization, x, weights, y.r0Implementation of forward propagation algorithm.—+Makes one forward step for the given layer.s1Implementation of backward propagation algorithm.˜,Makes one backward step for the given layer.t0Returns number of outputs of the given topology.™GReturns dimensions of weight matrices for given neural network topologyu©Create and initialize weights vector with random values for given neural network topology. Takes a seed to initialize generator of random numbers as a first parameter.vkCreate and initialize weights vector with random values for given neural network topology inside IO Monad.wnCreate and initialize weights vector with random values for given neural network topology inside RandomMonad.šeCreate and initialize list of weights matrices with random values for given neural network topology.x%Flatten list of matrices into vector.yIUnflatten vector into list of matrices for given neural network topology. nopqrstuvwxy nopqrstuvwxyn›Topology Maker(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneÝ­‚UCreates toplogy. Takes number of inputs, number of outputs and list of hidden layers. z{|}~€‚ ~€z{|}‚z{|}~€Neural Network(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneámƒONeural Network Model. Takes neural network topology as a constructor argument.œ3Score function. Takes a topology, X and theta list.4nuvwƒ„ƒ„4nuvwƒ„ .Multiclass Support Vector Machines Classifier."(c) Alexander Ignatyev, 2017-2018.BSD-3 experimentalPOSIXNoneåD†`Multiclass SVM Classifier, takes delta and number of futures. Delta = 1.0 is good for all cases.CDEFGHIJKLM†‡†‡†‡"#$%&'()*+,-./01234546789:4 ; < = > ? @ A B C D E F G H I 4J4KLMNOPQRS!TUVWXWXYZ[\]^_`abcdefghijklmnopqrstuvJ4KJ4Kwwxyz{|}~~€‚ƒ„…†K‡ˆ‰Š‹ŒŽ‘’“”•–†—˜™ š › œ žŸ ¡¢£¤¥!¦§¨©ª«¬­®„¯°%mltool-0.2.0.0-Ey8MRUAZb684uDjlY71NDk,MachineLearning.NeuralNetwork.TanhActivationMachineLearning.TypesMachineLearning.RegularizationMachineLearning.Random2MachineLearning.NeuralNetwork.WeightInitialization,MachineLearning.NeuralNetwork.ReluActivation,MachineLearning.NeuralNetwork.RegularizationMachineLearning.Model5MachineLearning.Optimization.MinibatchGradientDescent,MachineLearning.Optimization.GradientDescentMachineLearning.OptimizationMachineLearning.LogisticModel/MachineLearning.NeuralNetwork.SigmoidActivation*MachineLearning.NeuralNetwork.LogisticLoss!MachineLearning.LeastSquaresModelMachineLearning.RegressionMachineLearning.Clustering'MachineLearning.Classification.OneVsAll MachineLearning.TerminalProgress%MachineLearning.Classification.BinaryMachineLearningMachineLearning.PCA)MachineLearning.Classification.MultiClassMachineLearning.Utils!MachineLearning.SoftmaxClassifier)MachineLearning.NeuralNetwork.SoftmaxLoss*MachineLearning.NeuralNetwork.MultiSvmLoss#MachineLearning.NeuralNetwork.Layer&MachineLearning.NeuralNetwork.Topology+MachineLearning.NeuralNetwork.TopologyMakerMachineLearning.NeuralNetwork"MachineLearning.MultiSvmClassifier'MachineLearning.Classification.Internalbase GHC.FloattanhMatrixVectorRRegularizationRegNoneL2costReg gradientRegsamplesampleMgetRandomRListMgetRandomRVectorMgetRandomRMatrixMnguyenhegradientrelu forwardReg backwardRegModel hypothesiscostminibatchGradientDescentgradientDescentMinimizeMethodGradientDescentMinibatchGradientDescentConjugateGradientFRConjugateGradientPRBFGS2minimize checkGradient LogisticModelLogisticsigmoidsigmoidGradient$fModelLogisticModelscoreslossLeastSquaresModel LeastSquares$fModelLeastSquaresModelnormalEquationnormalEquation_pCluster kmeansIterMkmeans calcAccuracylearnWithProgressBarlearnOneVsAllWithProgressBarpredictlearnaddBiasDimensionremoveBiasDimension meanStddevfeatureNormalization mapFeatures splitToXY getDimReducergetDimReducer_rvMultiClassModel MultiClass Classifiercscore chypothesisccost cgradient cnumClasses processOutputccostReg cgradientReg$fModelMultiClassModel reduceByRowsVreduceByColumnsV reduceByRowsreduceByColumns sumByColumns sumByRowslistOfTuplesToListSoftmaxClassifierSoftmax$fClassifierSoftmaxClassifierLayerlUnitslForward lBackward lActivationlActivationGradientlInitializeThetaMCachecacheZcacheXcacheW affineForwardaffineBackwardTopologyLossFunc makeTopologypropagateForwardpropagateBackward numberOutputsinitializeThetainitializeThetaIOinitializeThetaMflatten unflattenLoss LLogisticLSoftmax LMultiSvm ActivationASigmoidAReluATanhNeuralNetworkModel NeuralNetwork$fModelNeuralNetworkModelMultiSvmClassifierMultiSvm$fClassifierMultiSvmClassifierrandomsInRangesMminibatchGradientDescentMnearestCentroidIndexcalcClusterCostcalcCostgetNewCentroid moveCentroidsbuildClusterList kmeansIterprocessOutputOneVsAllbuildOptPathMatrixcovarianceMatrixpcasvmD forwardPass backwardPass getThetaSizesinitializeThetaListM calcScores