Programming Assignment_Classification.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "This is the second assignment for the Coursera course \"Advanced Machine Learning and Signal Processing\"\n\n\nJust execute all cells one after the other and you are done - just note that in the last one you have to update your email address (the one you've used for coursera) and obtain a submission token, you get this from the programming assignment directly on coursera.\n\nPlease fill in the sections labelled with \"###YOUR_CODE_GOES_HERE###\""
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n\nIn case you are facing issues, please read the following two documents first:\n\nhttps://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n\nhttps://github.com/IBM/skillsnetwork/wiki/FAQ\n\nThen, please feel free to ask:\n\nhttps://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n\nPlease make sure to follow the guidelines before asking a question:\n\nhttps://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n\n\nIf running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/markdown": "# <span style=\"color:red\"><<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>></span>",
                        "text/plain": "<IPython.core.display.Markdown object>"
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": "from IPython.display import Markdown, display\ndef printmd(string):\n    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n\n\nif ('sc' in locals() or 'sc' in globals()):\n    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": "#!pip install pyspark==2.4.5"
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": "try:\n    from pyspark import SparkContext, SparkConf\n    from pyspark.sql import SparkSession\nexcept ImportError as e:\n    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n\nspark = SparkSession \\\n    .builder \\\n    .getOrCreate()"
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "--2023-02-21 16:14:46--  https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet\nResolving github.com (github.com)... 140.82.112.4\nConnecting to github.com (github.com)|140.82.112.4|:443... connected.\nHTTP request sent, awaiting response... 301 Moved Permanently\nLocation: https://github.com/claimed-framework/component-library/raw/master/coursera_ml/a2.parquet [following]\n--2023-02-21 16:14:46--  https://github.com/claimed-framework/component-library/raw/master/coursera_ml/a2.parquet\nReusing existing connection to github.com:443.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ml/a2.parquet [following]\n--2023-02-21 16:14:46--  https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ml/a2.parquet\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 59032 (58K) [application/octet-stream]\nSaving to: \u2018a2.parquet.2\u2019\n\na2.parquet.2        100%[===================>]  57.65K  --.-KB/s    in 0.001s  \n\n2023-02-21 16:14:47 (67.2 MB/s) - \u2018a2.parquet.2\u2019 saved [59032/59032]\n\n"
                }
            ],
            "source": "!wget https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Now it\u2019s time to have a look at the recorded sensor data. You should see data similar to the one exemplified below\u2026.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "+-----+-----------+-------------------+-------------------+-------------------+\n|CLASS|   SENSORID|                  X|                  Y|                  Z|\n+-----+-----------+-------------------+-------------------+-------------------+\n|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|\n|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|\n|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|\n|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|\n|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|\n|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|\n|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|\n|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|\n|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|\n|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|\n|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|\n|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|\n|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|\n|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|\n|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|\n|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|\n|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|\n|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|\n|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|\n|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|\n+-----+-----------+-------------------+-------------------+-------------------+\nonly showing top 20 rows\n\n"
                }
            ],
            "source": "df=spark.read.load('a2.parquet')\n\ndf.createOrReplaceTempView(\"df\")\nspark.sql(\"SELECT * from df\").show()\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please create a VectorAssembler which consumes columns X, Y and Z and produces a column \u201cfeatures\u201d\n"
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [],
            "source": "from pyspark.ml.feature import VectorAssembler\nvectorAssembler = VectorAssembler(inputCols=[\"X\", \"Y\", \"Z\"], outputCol=\"features\")"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please instantiate a classifier from the SparkML package and assign it to the classifier variable. Make sure to either\n1.\tRename the \u201cCLASS\u201d column to \u201clabel\u201d or\n2.\tSpecify the label-column correctly to be \u201cCLASS\u201d\n"
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [],
            "source": "from pyspark.ml.classification import GBTClassifier \n\nclassifier = GBTClassifier(labelCol='CLASS', featuresCol='features', maxIter=10)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Let\u2019s train and evaluate\u2026\n"
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [],
            "source": "from pyspark.ml import Pipeline\npipeline = Pipeline(stages=[vectorAssembler, classifier])"
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [],
            "source": "model = pipeline.fit(df)"
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [],
            "source": "prediction = model.transform(df)"
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\n|CLASS|   SENSORID|                  X|                  Y|                  Z|            features|       rawPrediction|         probability|prediction|\n+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\n|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|[380.664340054951...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|[104.743242992096...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|[118.114692361299...|[1.32680169324328...|[0.93423273625928...|       0.0|\n|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|[246.553940306425...|[1.32711813715030...|[0.93427161142354...|       0.0|\n|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|[-190.32584900181...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|[178.623963823874...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|[85.0312880518949...|[1.32621410603712...|[0.93416049441777...|       0.0|\n|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|[26.7862626747365...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|[-16.203752396859...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|[47.2048142440404...|[1.32589955669978...|[0.93412179134479...|       0.0|\n|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|[15.6088723989392...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|[-4.8281721129789...|[1.32396579485090...|[0.93388339065422...|       0.0|\n|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|[-98.407257128527...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|[22.8358453948165...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|[84.2017807008032...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|[56.5473252134512...|[1.32589766213010...|[0.93412155816733...|       0.0|\n|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|[-57.600865524774...|[1.33122144395126...|[0.93477377205806...|       0.0|\n|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|[-65.592647383890...|[1.32590267922033...|[0.93412217565278...|       0.0|\n|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|[34.8233735129100...|[1.51228073545265...|[0.95367147883022...|       0.0|\n|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|[39.8057382343912...|[1.32711813715030...|[0.93427161142354...|       0.0|\n+-----+-----------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+----------+\nonly showing top 20 rows\n\n"
                }
            ],
            "source": "prediction.show()"
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": "0.9986850756081526"
                    },
                    "execution_count": 13,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\nbinEval = MulticlassClassificationEvaluator().setMetricName(\"accuracy\") .setPredictionCol(\"prediction\").setLabelCol(\"CLASS\")\n    \nbinEval.evaluate(prediction) "
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "If you are happy with the result (I\u2019m happy with > 0.55) please submit your solution to the grader by executing the following cells, please don\u2019t forget to obtain an assignment submission token (secret) from the Coursera\u2019s graders web page and paste it to the \u201csecret\u201d variable below, including your email address you\u2019ve used for Coursera. (0.55 means that you are performing better than random guesses)\n"
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [],
            "source": "!rm -Rf a2_m2.json"
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "metadata": {},
            "outputs": [],
            "source": "prediction = prediction.repartition(1)\nprediction.write.json('a2_m2.json')"
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "--2023-02-21 16:17:41--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 2540 (2.5K) [text/plain]\nSaving to: \u2018rklib.py\u2019\n\nrklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n\n2023-02-21 16:17:42 (37.9 MB/s) - \u2018rklib.py\u2019 saved [2540/2540]\n\n"
                }
            ],
            "source": "!rm -f rklib.py\n!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [],
            "source": "import zipfile\nimport os\n\ndef zipdir(path, ziph):\n    for root, dirs, files in os.walk(path):\n        for file in files:\n            ziph.write(os.path.join(root, file))\n\nzipf = zipfile.ZipFile('a2_m2.json.zip', 'w', zipfile.ZIP_DEFLATED)\nzipdir('a2_m2.json', zipf)\nzipf.close()"
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "metadata": {},
            "outputs": [],
            "source": "!base64 a2_m2.json.zip > a2_m2.json.zip.base64"
        },
        {
            "cell_type": "code",
            "execution_count": 19,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "Submission successful, please check on the coursera grader page for the status\n-------------------------\n{\"elements\":[{\"itemId\":\"LTL4F\",\"id\":\"f_F-qCtuEei_fRLwaVDk3g~LTL4F~RYDThrIDEe2VphICt6NGUQ\",\"courseId\":\"f_F-qCtuEei_fRLwaVDk3g\"}],\"paging\":{},\"linked\":{}}\n-------------------------\n"
                }
            ],
            "source": "from rklib import submit\nkey = \"J3sDL2J8EeiaXhILFWw2-g\"\npart = \"G4P6f\"\nemail = \"mdanasmondol43@gmail.com\"\ntoken = \"Bs05o1Gyan00dGP1\" # (have a look here if you need more information on how to obtain the token https://youtu.be/GcDo0Rwe06U?t=276)\n\nwith open('a2_m2.json.zip.base64', 'r') as myfile:\n    data=myfile.read()\nsubmit(email, token, key, part, [part], data)"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.9 with Spark",
            "language": "python3",
            "name": "python39"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.9.13"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}