{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "The idea behind nornir_tests is to allow for adding tests against the task run and its results.  Many times, a task can run and get some output but that doesn't mean it is successful.  One option is to add additional checks or tasks with some conditional logic to deal with this.  Another is to add tests against the data returned from a task.  That is what nornir_tests is for.\n",
    "\n",
    "The question is probably why use tests.  Tests were inspired by the same feature in postman.  In postman, a test was something that was done against the data returned from a postman request to determine if the request was actually successful.  This can certainly be added in python using either pure python or additional tasks but there is some elegance and some additional functionality obtained by using tests tied to tasks.  In this intro, I will try to touch on a variety of ideas using tests.  In further tutorials I'll go into more detail on using each of the test types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nornir initialization\n",
    "\n",
    "This is standard nornir initialization and imports of tasks.  A different library nornir_rich is used for output to provide more detail and highlighting.  Using print_result is also possible if vars is set to include 'tests' attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nornir_napalm.plugins.tasks import napalm_get, napalm_ping\n",
    "from nornir_netmiko.tasks import netmiko_send_command\n",
    "from nornir import InitNornir\n",
    "\n",
    "nr = InitNornir(\n",
    "    inventory={\n",
    "        \"plugin\": \"SimpleInventory\",\n",
    "        \"options\": {\n",
    "            \"host_file\": \"data/hosts.yaml\",\n",
    "            \"group_file\": \"data/groups.yaml\",\n",
    "            \"defaults_file\": \"data/defaults.yaml\",\n",
    "        },\n",
    "    },\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports of nornir_tests libraries\n",
    "\n",
    "The print_result in nornir_utils works fine if 'tests' is added to vars.  There is also a modified version of print_result included in nornir_tests that provides a bit better reporting of test records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from nornir_tests.plugins.tests import regexp, until, timing, jpath\n",
    "from nornir_tests.plugins.tasks import wrap_task\n",
    "from nornir_tests.plugins.functions import print_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run first wrapped task\n",
    "\n",
    "In the first example, a vyos router interface facts are obtained using napalm and a test is run to validate the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\u001b[1m\u001b[36mnapalm_get**********************************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[34m* vyos ** changed : False ******************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[32mvvvv napalm_get ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO\u001b[0m\n\u001b[0m{\u001b[0m\u001b[0m'interfaces'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'eth0'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'description'\u001b[0m:\u001b[0m''\u001b[0m,\n\u001b[0m'is_enabled'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'is_up'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'last_flapped'\u001b[0m:\u001b[0m-1.0\u001b[0m,\n\u001b[0m'mac_address'\u001b[0m:\u001b[0m'08:00:27:e0:28:63'\u001b[0m,\n\u001b[0m'mtu'\u001b[0m:\u001b[0m-1\u001b[0m,\n\u001b[0m'speed'\u001b[0m:\u001b[0m0\u001b[0m}\u001b[0m,\n\u001b[0m'lo'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'description'\u001b[0m:\u001b[0m''\u001b[0m,\n\u001b[0m'is_enabled'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'is_up'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'last_flapped'\u001b[0m:\u001b[0m-1.0\u001b[0m,\n\u001b[0m'mac_address'\u001b[0m:\u001b[0m'00:00:00:00:00:00'\u001b[0m,\n\u001b[0m'mtu'\u001b[0m:\u001b[0m-1\u001b[0m,\n\u001b[0m'speed'\u001b[0m:\u001b[0m0\u001b[0m}\u001b[0m}\u001b[0m}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32mP JpathRecord - {'assertion': 'is_true',\n 'path': 'interfaces.eth0.is_up',\n 'result_attr': 'result'}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32m{'matches': ['interfaces.eth0.is_up']}\u001b[0m\n\u001b[0m\u001b[1m\u001b[32m^^^^ END napalm_get ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\u001b[0m"
    }
   ],
   "source": [
    "vyos = nr.filter(name='vyos')\n",
    "\n",
    "result = vyos.run(\n",
    "    task=wrap_task(napalm_get),\n",
    "    getters=['interfaces'],\n",
    "    tests=[\n",
    "        jpath(path='interfaces.eth0.is_up', assertion='is_true'),\n",
    "    ]\n",
    ")\n",
    "\n",
    "print_result(result, vars=['result', 'tests'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run ping with timing requirements\n",
    "\n",
    "Next a ping is sent to the vyos device using the timing test.  As can be seen in the output, this gives runtime of the task and can be enhanced with min_run_time and max_run_time to affect the task result if the fail_task argument is used.  If a failure occurs in the test and fail_task is set to True, then the task will be marked as failed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\u001b[1m\u001b[36mnapalm_ping*********************************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[34m* vyos ** changed : False ******************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[32mvvvv napalm_ping ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO\u001b[0m\n\u001b[0m{\u001b[0m\u001b[0m'success'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'packet_loss'\u001b[0m:\u001b[0m0\u001b[0m,\n\u001b[0m'probes_sent'\u001b[0m:\u001b[0m5\u001b[0m,\n\u001b[0m'results'\u001b[0m:\u001b[0m[{'ip_address': '192.168.99.1', 'rtt': 4.288}]\u001b[0m,\n\u001b[0m'rtt_avg'\u001b[0m:\u001b[0m4.288\u001b[0m,\n\u001b[0m'rtt_max'\u001b[0m:\u001b[0m6.675\u001b[0m,\n\u001b[0m'rtt_min'\u001b[0m:\u001b[0m3.434\u001b[0m,\n\u001b[0m'rtt_stddev'\u001b[0m:\u001b[0m1.216\u001b[0m}\u001b[0m}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32mP TimingRecord - {'max_run_time': 9223372036854775807}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32m{'run_time': 4.914522171020508,\n 't0': 1600801169.7141054,\n 't1': 1600801174.6286275}\u001b[0m\n\u001b[0m\u001b[1m\u001b[32m^^^^ END napalm_ping ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\u001b[0m"
    }
   ],
   "source": [
    "result = vyos.run(\n",
    "    task=wrap_task(napalm_ping),\n",
    "    dest='192.168.99.1',\n",
    "    tests=[\n",
    "        timing(),\n",
    "    ]\n",
    ")\n",
    "print_result(result, vars=['result', 'tests'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get unfailed task to fail due to assertion\n",
    "\n",
    "The next example shows how the result failed state can be affected by a test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\u001b[1m\u001b[36mnapalm_ping*********************************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[34m* vyos ** changed : False ******************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[31mvvvv napalm_ping ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ERROR\u001b[0m\n\u001b[0m{\u001b[0m\u001b[0m'success'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'packet_loss'\u001b[0m:\u001b[0m0\u001b[0m,\n\u001b[0m'probes_sent'\u001b[0m:\u001b[0m5\u001b[0m,\n\u001b[0m'results'\u001b[0m:\u001b[0m[{'ip_address': '192.168.99.1', 'rtt': 3.342}]\u001b[0m,\n\u001b[0m'rtt_avg'\u001b[0m:\u001b[0m3.342\u001b[0m,\n\u001b[0m'rtt_max'\u001b[0m:\u001b[0m4.664\u001b[0m,\n\u001b[0m'rtt_min'\u001b[0m:\u001b[0m2.631\u001b[0m,\n\u001b[0m'rtt_stddev'\u001b[0m:\u001b[0m0.696\u001b[0m}\u001b[0m}\u001b[0m\n\u001b[0m\u001b[2m\u001b[31mF TimingRecord - {'fail_task': True, 'max_run_time': 1}\u001b[0m\n\u001b[0m\u001b[2m\u001b[31m{'run_time': 4.914187431335449,\n 't0': 1600801174.8475542,\n 't1': 1600801179.7617416}\u001b[0m\n\u001b[0m\u001b[1m\u001b[31m^^^^ END napalm_ping ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\u001b[0m"
    }
   ],
   "source": [
    "result = vyos.run(\n",
    "    task=wrap_task(napalm_ping),\n",
    "    dest='192.168.99.1',\n",
    "    tests=[\n",
    "        timing(max_run_time=1, fail_task=True),\n",
    "    ]\n",
    ")\n",
    "print_result(result, vars=['result', 'tests'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining tests\n",
    "\n",
    "Tests can be stacked and the entire task and associated tests can be retried using until.  The next test shows how a rebooted device can be retried until it comes back up using only tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\u001b[1m\u001b[36mnetmiko_send_command************************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[34m* vyos ** changed : False ******************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[32mvvvv netmiko_send_command ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO\u001b[0m\n\u001b[0m\u001b[1m\u001b[32m^^^^ END netmiko_send_command ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\u001b[0m\u001b[1m\u001b[36mnapalm_get**********************************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[34m* vyos ** changed : False ******************************************************\u001b[0m\n\u001b[0m\u001b[1m\u001b[32mvvvv napalm_get ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO\u001b[0m\n\u001b[0m{\u001b[0m\u001b[0m'interfaces'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'eth0'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'description'\u001b[0m:\u001b[0m''\u001b[0m,\n\u001b[0m'is_enabled'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'is_up'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'last_flapped'\u001b[0m:\u001b[0m-1.0\u001b[0m,\n\u001b[0m'mac_address'\u001b[0m:\u001b[0m'08:00:27:e0:28:63'\u001b[0m,\n\u001b[0m'mtu'\u001b[0m:\u001b[0m-1\u001b[0m,\n\u001b[0m'speed'\u001b[0m:\u001b[0m0\u001b[0m}\u001b[0m,\n\u001b[0m'lo'\u001b[0m:\u001b[0m{\u001b[0m\u001b[0m'description'\u001b[0m:\u001b[0m''\u001b[0m,\n\u001b[0m'is_enabled'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'is_up'\u001b[0m:\u001b[0mTrue\u001b[0m,\n\u001b[0m'last_flapped'\u001b[0m:\u001b[0m-1.0\u001b[0m,\n\u001b[0m'mac_address'\u001b[0m:\u001b[0m'00:00:00:00:00:00'\u001b[0m,\n\u001b[0m'mtu'\u001b[0m:\u001b[0m-1\u001b[0m,\n\u001b[0m'speed'\u001b[0m:\u001b[0m0\u001b[0m}\u001b[0m}\u001b[0m}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32mP JpathRecord - {'assertion': 'is_true',\n 'fail_task': True,\n 'path': 'interfaces.eth0.is_up',\n 'result_attr': 'result'}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32m{'matches': ['interfaces.eth0.is_up']}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32mP UntilRecord - {'delay': 15, 'initial_delay': 15, 'reset_conns': True, 'retries': 10}\u001b[0m\n\u001b[0m\u001b[2m\u001b[32m{'run_time': 56.76758694648743,\n 't0': 1600801185.3881028,\n 't1': 1600801242.1556897}\u001b[0m\n\u001b[0m\u001b[1m\u001b[32m^^^^ END napalm_get ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\u001b[0m"
    }
   ],
   "source": [
    "nr.data.reset_failed_hosts()\n",
    "\n",
    "print_result(\n",
    "    vyos.run(\n",
    "        task=netmiko_send_command,\n",
    "        command_string='reboot now'\n",
    "    )\n",
    ")\n",
    "\n",
    "result = vyos.run(\n",
    "    task=wrap_task(napalm_get),\n",
    "    getters=['interfaces'],\n",
    "    tests=[\n",
    "        jpath(path='interfaces.eth0.is_up', assertion='is_true', fail_task=True),\n",
    "        until(initial_delay=15, retries=10, delay=15, reset_conns=True),\n",
    "    ]\n",
    ")\n",
    "print_result(result, vars=['result', 'tests'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result above shows that it must have failed on its first try but around the one minute mark it was able to pass the jpath test and so until finished up.\n",
    "\n",
    "Check out all the other tutorials to see each in more detail."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.2 64-bit ('.venv': venv)",
   "language": "python",
   "name": "python38264bitvenvvenv1c35bb63d8ff4e6aa6d1792b190afa1b"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}