>        ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ! Common Types(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneMatrix Types (of Doubles)Vector Types (of Doubles)Scalar Type (Double)Utils(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone "Converts list of tuples into list.     Regularization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone Regularization No regularization L2 Regularization`Calculates regularization for Model.cost function. It takes regularization parameter and theta.dCalculates regularization for Model.gradient function. It takes regularization parameter and theta.     $Random generation utility functions.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneSamples n+ (given as a second parameter) values from list (given as a third parameter).Samples n+ (given as a second parameter) values from list1 (given as a third parameter) inside RandomMonad.AReturns a list of random values distributed in a closed interval rangeCReturns a vector of random values distributed in a closed interval rangeCReturns a matrix of random values distributed in a closed interval rangeTakes a list of ranges `(lo, hi)`, returns a list of random values uniformly distributed in the list of closed intervals [(lo, hi)].list's lengthsrange(list of random values inside RandomMonadvector's lengthrangenumber of rowsnumber of columnsrangeWeight Initialization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone<Weight Initialization Algorithm discussed in Nguyen at al.:  :https://web.stanford.edu/class/ee373b/nninitialization.pdf Nguyen, Derrick, Widrow, B. Improving the learning speed of 2-layer neural networks by choosing initial values of adaptive weights. In Proc. IJCNN, 1990; 3: 21-26.8Weight Initialization Algorithm discussed in He at al.:  https://arxiv.org/abs/1502.01852 Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.Tanh Activation.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone Softmax Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneReLu Activation(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone Regularization(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNonefCalcaulates regularization for forward propagation. It takes regularization parameter and theta list.iCalculates regularization for step of backward propagation. It takes regularization parameter and theta.   Multi SVM Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone SVM Delta ! ! ! ! Neural Network's Layer(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneOT"#$%&'()*+,-./0"#$%&'()*+,-./0"#$%&'()*+,-./0"#$%&'()*+,-./0 Neural Network's Topology(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone1Neural network topology has at least 2 elements: numver of input and number of outputs. And sizes of hidden layers between 2 elements.2.Loss function's type. Takes x, weights and y.3pMakes Neural Network's Topology. Takes number of inputs, list of hidden layers, output layer and loss function.4WCalculates loss for the given topology. Takes topology, regularization, x, weights, y.50Implementation of forward propagation algorithm.+Makes one forward step for the given layer.61Implementation of backward propagation algorithm.,Makes one backward step for the given layer.70Returns number of outputs of the given topology.GReturns dimensions of weight matrices for given neural network topology8Create and initialize weights vector with random values for given neural network topology. Takes a seed to initialize generator of random numbers as a first parameter.9kCreate and initialize weights vector with random values for given neural network topology inside IO Monad.:nCreate and initialize weights vector with random values for given neural network topology inside RandomMonad.eCreate and initialize list of weights matrices with random values for given neural network topology.;%Flatten list of matrices into vector.<IUnflatten vector into list of matrices for given neural network topology.123456789:;< 123456789:;< 123456789:;<123456789:;< Regression Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone>{Hypothesis function, a.k.a. score function (for lassifition problem) Takes X (m x n) and theta (n x 1), returns y (m x 1).?Cost function J(Theta), a.k.a. loss function. It takes regularizarion parameter, matrix X (m x n), vector y (m x 1) and vector theta (n x 1).@Gradient function. It takes regularizarion parameter, X (m x n), y (m x 1) and theta (n x 1). Returns vector of gradients (n x 1).=>?@=?@>=>?@=>?@Gradient Descent(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneA,Gradient Descent method implementation. See MachineLearning.Regression for usage details.AAAAGradient Descent(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneB6Minibatch Gradient Descent method implementation. See MachineLearning.Regression for usage details.6Minibatch Gradient Descent method implementation. See MachineLearning.Regression for usage details.B seed batch sizelearning rate, alphamodel to learnepsilonmax number of iters regularization parameter, lambdamatrix of features, Xoutput vector, y %vector of initial weights, theta or w $vector of weights and learning path  batch sizelearning rate, alphamodel to learnepsilonmax number of iters regularization parameter, lambdamatrix of features, Xoutput vector, y%vector of initial weights, theta or w $vector of weights and learning path BBB Optimization(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneD>Gradient descent, takes alpha. Requires feature normalization.E<Minibacth Gradietn Descent, takes seed, batch size and alphaFrFletcher-Reeves conjugate gradient algorithm, takes size of first trial step (0.1 is fine) and tol (0.1 is fine).GpPolak-Ribiere conjugate gradient algorithm. takes size of first trial step (0.1 is fine) and tol (0.1 is fine).HwBroyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, takes size of first trial step (0.1 is fine) and tol (0.1 is fine).IReturns solution vector (theta) and optimization path. Optimization path's row format: [iter number, cost function value, theta values...]J,Gradient checking function. Approximates the derivates of the Model's cost function and calculates derivatives using the Model's gradient functions. Returns norn_2 between 2 derivatives. Takes model, regularization, X, y, theta and epsilon (used to approximate derivatives, 1e-4 is a good value). CDEFGHI.model (Least Squares, Logistic Regression etc)*epsilon, desired precision of the solution$maximum number of iterations allowedregularization parameterXyinitial solution, theta%solution vector and optimization pathJCDEFGHIJCDEFGHIJCDEFGHIJLogistic Regression Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneMCalculates sigmoidN!Calculates derivatives of sigmoidKLMNO=?@>KLMNKLMNKLMNOLogistic Loss.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNonePQRPQRPQRPQRSigmoid(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneSMSMSSTopology Maker(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone\UCreates toplogy. Takes number of inputs, number of outputs and list of hidden layers.TUVWXYZ[\ TUVWX[YZ\ XYZ[TUVW\ TUVWXYZ[\Least Squares Model(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNone]^_]^]^]^_ Regression!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNone`gNormal equation using inverse, does not require feature normalization It takes X and y, returns theta.afNormal equation using pseudo inverse, requires feature normalization It takes X and y, returns theta.`a =?@>CDEFGHI]^`a=>?@]^CDEFGHI`a `a Clustering(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNone b;Cluster type (list of samples associtaed with the cluster).0Gets list if the nearest centroid to the sample.,Calculates cost associated with one claster.'Calculates cost value for all clusters.8Calculates centroid (middle point) of the given cluster.2Calculates new cluster centroids for each cluster.1Build cluster list from list of clusters indices.c/Run K-Means algorithm once inside Random Monad.dSClusters data using K-Means Algorithm inside Random Monad. Runs K-Means algorithm N1 times, returns the clustering with smaller cost. blist of cluster cetroids;sample;index of the nearest centroid.cluster;cluster centroid; cost value. cluster list;list of cluster centroids; cost value.cluster; centroid.list of clusters;list of cluster centroids.list of samples;Clist of cluster indices (associated cluster index for each sample);list of clusters.list of samples;number of clusters (K);list of initial centroids; (list of clusters, cost values).clist of samples;number of clusters (K);iteration number;4(list of clusters, cost values) inside Random Monad.d"number of K-Means Algorithm runs (N);data to cluster;desired number of clusters (K);%list of clusters inside Random Monad.bcdbdc bcd!Classification Internal module.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNoneeCalculates accuracy of Classification predictions. Takes vector expected y and vector predicted y. Returns number from 0 to 1, the closer to 1 the better accuracy. Suitable for both Classification Types: Binary and Multiclass.Process outputs for One-vs-All Classification. Takes number of labels and output vector y. Returns list of vectors of binary outputs (One-vs-All Classification). It is supposed that labels are integerets start at 0.eeeOne-vs-All Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNonefOne-vs-All Classification prediction function. Takes a matrix of features X and a list of vectors theta, returns predicted class number assuming that class numbers start at 0.g Learns One-vs-All Classificationfg (e.g. BFGS2 0.1 0.1)+epsilon, desired precision of the solution;%maximum number of iterations allowed; regularization parameter lambda;number of labels matrix X;vector yinitial theta list;&solution vector and optimization path. CDEFGHefg CDEFGHfge fgNeural Network(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNonehONeural Network Model. Takes neural network topology as a constructor argument.3Score function. Takes a topology, X and theta list.hij 189:=?@>ehi=>?@hie189: hij.Learn function with progress bar for terminal.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNonekLearn the given function displaying progress bar in terminal. It takes function, initial theta and number of iterations to call the function. It returns theta and optimization path (see MachineLearning.Optimization for details).lLearn the given function displaying progress bar in terminal. It takes function, list of outputs and list of initial thetas and number of iterations to call the function. It returns list of thetas and list of optimization paths (see MachineLearning.Optimization for details).PBuild a single optimazation path matrix from list of optimization path matrices.klklklklBinary Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNonemvBinary Classification prediction function. Takes a matrix of features X and a vector theta. Returns predicted class.nLearns Binary Classification.mn(e.g. BFGS2 0.1 0.1)+epsilon, desired precision of the solution;%maximum number of iterations allowed; regularization parameter lambda; matrix X;binary vector y;initial Theta;&solution vector and optimization path. CDEFGHemn CDEFGHmne mnMachine Learning(c) Alexander Ignatyev, 2016BSD-3 experimentalPOSIXNoneo'Add biad dimension to the future matrixpRemove biad dimensionqzCaclulates mean and stddev values of every feature. Takes feature matrix X, returns pair of vectors of means and stddevs.sJMaps the features into all polynomial terms of X up to the degree-th powert?Splits data matrix to features matrix X and vector of outputs yopqrstopqrstopqrstopqrstMultiClass Classification.!(c) Alexander Ignatyev, 2016-2017BSD-3 experimentalPOSIXNone u8MultiClassModel is Model wrapper class around ClassifierwCClassifier type class represents Multi-class classification models.xScore functionyJHypothesis function Takes X (m x n) and theta (n x k), returns y (m x k).zCost function J(Theta), a.k.a. loss function. It takes regularizarion parameter lambda, matrix X (m x n), vector y (m x 1) and vector theta (n x 1).{Gradient function. It takes regularizarion parameter lambda, X (m x n), y (m x 1) and theta (n x 1). Returns vector of gradients (n x 1).|Returns Number of Classes}Process outputs for MultiClass Classification. Takes Classifier and output vector y. Returns matrix of binary outputs. It is supposed that labels are integerets start at 0.~]Calculates regularization for Classifier.ccost. It takes regularization parameter and theta.aCalculates regularization for Classifier.cgradient. It takes regularization parameter and theta. uvwxyz{|}~ uvwxyz{|}~wxyz{|uv} ~uvwxyz{|}~.Multiclass Support Vector Machines Classifier.(c) Alexander Ignatyev, 2017.BSD-3 experimentalPOSIXNone`Multiclass SVM Classifier, takes delta and number of futures. Delta = 1.0 is good for all cases. =?@>uvwxyz{|}~Principal Component Analysis.(c) Alexander Ignatyev, 2017BSD-3 experimentalPOSIXNoneComputes "covariance matrix".^Compute eigenvectors (matrix U) and singular values (matrix S) of the given covariance matrix.Gets dimensionality reduction function, retained variance (0..1) and reduced X for given matrix X and number of dimensions to retain.Gets dimensionality reduction function, retained number of dimensions and reduced X for given matrix X and variance to retain (0..1]. Softmax Classifier.(c) Alexander Ignatyev, 2017.BSD-3 experimentalPOSIXNone,Softmax Classifier, takes number of classes. =?@>uvwxyz{|}~"#$%&'()*+,-./0123456789:;<;=>; ? @ < ; = A A B C D E F G H H I J K L M N O P = Q R S T U V W X Y Z [ ;\]^_`abcdefghij<;=;klmnopqrPstuvwxyz!{|}~|}    N=!X %mltool-0.1.0.2-5Foql4ZjiJ5B382tyQjtxY,MachineLearning.NeuralNetwork.TanhActivationMachineLearning.TypesMachineLearning.UtilsMachineLearning.RegularizationMachineLearning.Random2MachineLearning.NeuralNetwork.WeightInitialization)MachineLearning.NeuralNetwork.SoftmaxLoss,MachineLearning.NeuralNetwork.ReluActivation,MachineLearning.NeuralNetwork.Regularization*MachineLearning.NeuralNetwork.MultiSvmLoss#MachineLearning.NeuralNetwork.Layer&MachineLearning.NeuralNetwork.TopologyMachineLearning.Model,MachineLearning.Optimization.GradientDescent5MachineLearning.Optimization.MinibatchGradientDescentMachineLearning.OptimizationMachineLearning.LogisticModel*MachineLearning.NeuralNetwork.LogisticLoss/MachineLearning.NeuralNetwork.SigmoidActivation+MachineLearning.NeuralNetwork.TopologyMaker!MachineLearning.LeastSquaresModelMachineLearning.RegressionMachineLearning.Clustering'MachineLearning.Classification.OneVsAllMachineLearning.NeuralNetwork MachineLearning.TerminalProgress%MachineLearning.Classification.BinaryMachineLearning)MachineLearning.Classification.MultiClass"MachineLearning.MultiSvmClassifierMachineLearning.PCA!MachineLearning.SoftmaxClassifier'MachineLearning.Classification.Internalbase GHC.FloattanhMatrixVectorR reduceByRowsVreduceByColumnsV reduceByRowsreduceByColumns sumByColumns sumByRowslistOfTuplesToListRegularizationRegNoneL2costReg gradientRegsamplesampleMgetRandomRListMgetRandomRVectorMgetRandomRMatrixMnguyenhegradientscoreslossrelu forwardReg backwardRegLayerlUnitslForward lBackward lActivationlActivationGradientlInitializeThetaMCachecacheZcacheXcacheW affineForwardaffineBackwardTopologyLossFunc makeTopologypropagateForwardpropagateBackward numberOutputsinitializeThetainitializeThetaIOinitializeThetaMflatten unflattenModel hypothesiscostgradientDescentminibatchGradientDescentMinimizeMethodGradientDescentMinibatchGradientDescentConjugateGradientFRConjugateGradientPRBFGS2minimize checkGradient LogisticModelLogisticsigmoidsigmoidGradient$fModelLogisticModelLoss LLogisticLSoftmax LMultiSvm ActivationASigmoidAReluATanhLeastSquaresModel LeastSquares$fModelLeastSquaresModelnormalEquationnormalEquation_pCluster kmeansIterMkmeans calcAccuracypredictlearnNeuralNetworkModel NeuralNetwork$fModelNeuralNetworkModellearnWithProgressBarlearnOneVsAllWithProgressBaraddBiasDimensionremoveBiasDimension meanStddevfeatureNormalization mapFeatures splitToXYMultiClassModel MultiClass Classifiercscore chypothesisccost cgradient cnumClasses processOutputccostReg cgradientReg$fModelMultiClassModelMultiSvmClassifierMultiSvm$fClassifierMultiSvmClassifier getDimReducergetDimReducer_rvSoftmaxClassifierSoftmax$fClassifierSoftmaxClassifierrandomsInRangesM tanhGradientsvmD forwardPass backwardPass getThetaSizesinitializeThetaListMminibatchGradientDescentM minimizeVD mkAffineLayer mkOutputLayerhiddenActivationhiddenGradientoutputActivationoutputGradientnearestCentroidIndexcalcClusterCostcalcCostgetNewCentroid moveCentroidsbuildClusterList kmeansIterprocessOutputOneVsAll calcScores processParamsbuildOptPathMatrixnewProgressBardoLoop learnOneClasspolynomialTermscombinationsWithReplacementcovarianceMatrixpcaremap