Merge pull request #3 from delphix/develop

Develop

Author: arunskurian

.gitignore

@@ -11,4 +11,7 @@ src/generated/*
 *.pyc
 *generated*
 .DS_Store
-__pycache__/

+__pycache__/
+*.iml
+artifact.json

+

README.md

@@ -1,4 +1,5 @@
-## 
+# MySQLLinux 
+
 ## What Does a Delphix Plugin Do?
 Delphix is a data management platform that provides the ability to securely copy and share datasets. Using virtualization, you will ingest your data sources and create virtual data copies, which are full read-write capable database instances that use a small fraction of the resources a normal database copy would require. The Delphix engine has built-in support for interfacing with certain types of datasets, such as Oracle, SQL Server and ASE.
 
@@ -7,105 +8,27 @@ The Delphix virtualization SDK (https://github.com/delphix/virtualization-sdk) p
 ## MySQL Plugin
 MySQL plugin is developed to virtualize MySQL data source leveraging the following built-in MySQL technologies:
 Features:
-  - Environment Discovery: MySQL plugin can discover environments where MySQL server is installed.
-  - Ingesting Data: Create a dSource using differnt methods specified below. 
-  - VDB Creation: Single node MySQL VDB can be provisioned from the dsource snapshot.
+- Environment Discovery: MySQL plugin can discover environments where MySQL server is installed.
+- Ingesting Data: Create a dSource using differnt methods specified below.
+- VDB Creation: Single node MySQL VDB can be provisioned from the dsource snapshot.
 
 Different Ways to Ingest Data ( Dsource creation )
-  - Replication with Delphix initiated Backup: Delphix takes an initial backup from source DB to ingest data and create a dSource. Delphix also sets up a master-slave replication to keep this dSource in sync with the source database. User can select the databases they want to virtualize
-  - Replication with User Provided Backup: User provides a backup file from source DB to ingest data and create a dSource. Delphix sets up a master-slave replication to keep this dSource in sync with your source database. 
-  - User Provided Backup with no Replication: User provides a backup file from source DB to ingest data and create a dSource. When a new backup is available, user initiates a resync of the dSource to ingest data from the new backup.
-  - Manual Backup Ingestion: Delphix creates an empty seed datanase and User manually ingests a backup to create a dSource.
-  - Simple Tablespace Backup/Subsetting: Allows to create a dSource using a specific list of tables from source database.
- 
-
-### Table of Contents
-1. [Prerequisites](#requirements-plugin)
-2. [Build and Upload Plugin](#upload-plugin)
-3. [Download logs](#run_unit_test_case)
-4. [Tested Versions](#tested-versions)
-5. [Supported Features](#support-features)
-6. [Unsupported Features](#unsupported-features)
-7. [How to Contribute](#contribute)
-8. [Statement of Support](#statement-of-support)
-9. [License](#license)
-
-
-### <a id="requirements-plugin"></a>Prerequisites
-**Software Requirements**
-1. jq - This toolkit requires jq to be configured on Staging and Target Hosts. 
-
-**MySQL instance requirements**
-1. Binary logging must be enabled on MySQL source instance.
-2. Server ID for the source must be greater than 0.
-    
-**MySQL database user with following privileges**
-1. delphixdb
-This MySQL user must be configured to have following privilege from the Delphix Engine IP, the staging host IP and localhost
-To grant the privilege for this user, use the following command:
-
-```js
-SQL> GRANT SELECT, SHUTDOWN, SUPER, RELOAD, REPLICATION CLIENT,REPLICATION SLAVE,SHOW VIEW, EVENT, TRIGGER on *.* to 'delphix'@'%';
-```
-
-OR
-
-```js
-SQL> GRANT ALL PRIVILEGES ON *.* TO '<delphix>'@'%';
-```
-
-#### _Staging Host Specific Requirements_
-
-**O/S user with following privileges**
-1. Regular o/s user. should be able to ps all processes.
-2. Should be in the same primary and secondary groups as mysql user ( or the MySQL binary owner )
-3. Execute access on all files within MySQL installation folder - Min permission level 775 recommended. 
-4. Sudo privileges for mount, umount. See sample below assuming `delphix_os` is used as delphix user.
-Example sudoers file entry
-```shell
-Defaults:delphix_os !requiretty
-delphix_os ALL=NOPASSWD: \ 
-/bin/mount, /bin/umount, /bin/mkdir, /bin/rmdir, /bin/ps
-```
-
-**Other Staging Host Requirements**
-
-1. Same version as Source MySQL Binaries installed.
-2. A MySQL config file (my.cnf) to be used for the Staging DB instance must be available under Delphix Toolkit Directory. 
-3. Empty folder on host to hold delphix toolkit  [ approximate 2GB free space ]
-4. Empty folder on host to mount nfs filesystem. This is just and empty folder with no space requirements and act as base folder for nfs mounts.
-
-
-#### _Target Requirements_
-
-**O/S user with following privileges**
-1. Regular o/s user. should be able to ps all processes.
-2. Should be in the same primary and secondary groups as mysql user ( or the MySQL binary owner )
-3. Execute access on all files within MySQL installation folder - Min permission level 775 recommended. 
-4. Sudo privileges for mount, umount. See sample below assuming `delphix_os` is used as delphix user.
-Example sudoers file entry
-```shell
-Defaults:delphix_os !requiretty
-delphix_os ALL=NOPASSWD: \ 
-/bin/mount, /bin/umount, /bin/mkdir, /bin/rmdir, /bin/ps
-```
-
-**Other Staging Host Requirements**
-
-1. Same version as Source MySQL Binaries installed.
-2. A MySQL config file (my.cnf) to be used for the Staging DB instance must be available under Delphix Toolkit Directory. 
-3. Empty folder on host to hold delphix toolkit  [ approximate 2GB free space ]
-4. Empty folder on host to mount nfs filesystem. This is just and empty folder with no space requirements and act as base folder for nfs mounts.
+- Replication with Delphix initiated Backup: Delphix takes an initial backup from source DB to ingest data and create a dSource. Delphix also sets up a master-slave replication to keep this dSource in sync with the source database. User can select the databases they want to virtualize
+- Replication with User Provided Backup: User provides a backup file from source DB to ingest data and create a dSource. Delphix sets up a master-slave replication to keep this dSource in sync with your source database.
+- User Provided Backup with no Replication: User provides a backup file from source DB to ingest data and create a dSource. When a new backup is available, user initiates a resync of the dSource to ingest data from the new backup.
+- Manual Backup Ingestion: Delphix creates an empty seed datanase and User manually ingests a backup to create a dSource.
 
+MySQL Plugin documentation can be found at https://delphix.github.io/mysqllinux/
 
 ### <a id="upload-plugin"></a>Steps to build, upload and run unit tests for plugin
 
-   1. Build the source code. It generates the build with name `artifacts.json`:
+1. Build the source code. It generates the build with name `artifacts.json`:
 ```bash
     dvp build
 ```
-    
-   2. Upload the `artifacts.json` ( generated in step 3 ) on Delphix Engine:
+  dvp version 2.1 required. 
+
+2. Upload the `artifacts.json` ( generated in step 1 ) on Delphix Engine:
 ```bash
     dvp upload -e <Delphix_Engine_Name> -u <username> --password <password>
 ```
@@ -120,19 +43,17 @@ Download the plugin logs using below command:
 
 ### <a id="tested-versions"></a>Tested Versions
 - Delphix Engine: 6.0.4.0 and above
-- MySQL Version: 5.7.7, 5.7.9, 5.7.10, 5.7.12   
-- Linux Version: RHEL 6.x
+- MySQL Versions 5.7.x above 5.7.6. 
+- Linux Version: RHEL 6.x, 7.x
 
 ### <a id="support-features"></a>Supported Features
 - MySQL Replication
-- MySQL Tablespace Hotbackup
 - MySQL Manual Backup Ingestion
 
 ### <a id="unsupported-features"></a>Unsupported Features
 - MySQL Clusters
 - Sharded MySQL Databases
 
-
 ### <a id="contribute"></a>How to Contribute
 
 Please read [CONTRIBUTING.md](./CONTRIBUTING.md) to understand the pull requests process.
@@ -144,4 +65,3 @@ This software is provided as-is, without warranty of any kind or commercial supp
 ### <a id="license"></a>License
 
 This is code is licensed under the Apache License 2.0. Full license is available [here](./LICENSE).
-

artifact.json

@@ -0,0 +1,15 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+mkdocs==0.17.5
+mkdocs-material==2.9.2
+markdown-include==0.5.1
+mkdocs-awesome-pages-plugin==1.2.0
+
+[requires]
+python_version = "3.7"
+
+[dev-packages]

test/__init__.py renamed to docs/.keep

@@ -0,0 +1 @@
+theme: jekyll-theme-minimal

docs/Pipfile

@@ -0,0 +1,8 @@
+arrange:
+  - index.md
+  - Pre-Requisites
+  - PluginInstallation.md
+  - Discovery.md
+  - Linking
+  - Provisioning.md
+  - Release_Notes

docs/_config.yml

@@ -0,0 +1,68 @@
+# Discovery
+
+Environment discovery/refresh is a process that enables the MySQL Plugin to determine MySQL installation details on a host. 
+Database discovery/refresh is initiated during the environment set up process.
+
+Whenever there is any change (installing a new database home) to an already set up environment in the Delphix application, 
+we need to perform an environment discovery/refresh.
+
+
+## Prerequisites
+
+-   Installation of the MySQL Plugin is required before the Discovery.
+
+## Refreshing an Environment
+
+Environment refresh will update the metadata associated with that environment and push Delphix Toolkit on to the host.
+
+1. Login to the **Delphix Management** application.
+2. Click **Manage**.
+3. Select **Environments**.
+4. In the Environments panel, click the name of the environment you want to refresh.
+5. Select the **Refresh** icon.
+6. In the Refresh confirmation dialog select **Refresh**.
+
+![Screenshot](./image/Image9.png)
+
+Once an environment refresh completes successfully, Delphix will discover all MySQL installations on the environment. 
+These installations are referred to as "repositories"
+
+
+## Add Source Config
+
+
+As noted avove, environments contain `repositories`, that are MySQL installations in the environment. 
+Each environment may have any number of repositories associated with it.  
+
+The next step in the virtualization process is to add a `SourceConfig`. 
+A `SourceConfig` object defines the configuratino of the dSource and is required to create a dSource.
+You can create any number of `SourceConfig` objects using a repository, which represent known database instances. 
+
+For the MySQL plugin, Source config is must be created manually.
+
+### How to create Source Config
+
+*Source Config is created on the Staging Environment*
+
+1. Login to the **Delphix Management** application.
+2. Click **Manage**.
+3. Select **Environments**.
+4. Select the repository.
+5. Click on **+** icon (Shown in next image).
+      ![Screenshot](./image/Image10.png)  
+
+6. Add required details in the `Add database` section.
+      
+      - Enter source data directory in section **Data Directory**.
+      - Enter source port number in **Port** section.
+      - Enter MySQL base directory on the source host in **Base Directory** section.
+      - Enter dsource name in **MySQL dSource Name** section.
+    
+      ![Screenshot](./image/Image11.png)
+
+
+## What's Next?
+
+
+Now that your environments are all added and a Source Config has been created, 
+please proceed to [Linking](/Linking/Replication_Mode/index.html) page to see how we can create the dSource.

docs/docs/.pages

@@ -0,0 +1,3 @@
+arrange:
+  - Replication_Mode.md
+  - Manual_Ingestion_Mode.md

docs/docs/Discovery.md

@@ -0,0 +1,82 @@
+# Manual Ingestion
+
+In Manual Ingestion Mode, Delphix creates a seed (empty) staging database. 
+Customer users will be responsible to manually ingest data into this staging db. 
+All other management operations can be performed through Delphix. A snapshot can be taken either manually 
+or via a SnapSync policy once the data is restored into the seed staging database. 
+
+### Pre-Requisites
+
+- Staging environment must be added to Delphix.
+- A Source Config must be created on the staging environment-MySQL repository.
+- <span class="code_title">[Recommended] </span>
+  As every  organization's MySQL configuration is different,
+  Delphix expects a starter *my.cnf* file to be present in Delphix Toolkit Directory when creating a staging database.
+  Delphix will use this *my.cnf* file and modify it as per the configuration provided during the the dsource creation process.
+
+  This is recommended to reduce the possibility of errors while restoring the backup from the source database.
+
+  !!! warning
+  It may take upto 5 minutes after successful dSource creation for the status to show as Active.
+
+
+### Creating dSource
+
+1. Login to **Delphix Management** application.
+2. Click **Manage** >  **Datasets**.
+3. Select **Add dSource**.
+
+   ![Screenshot](../../image/add-dsource.png)
+
+
+4. In the Add dSource wizard, select the MySQL source configuration which is created on the staging host.
+5. Select *Manual Ingestion* in the dSource Type dropdown.
+6. Provide the additional details required for dsource creation
+  - Staging DB Server ID
+
+    Server ID for the dsource (stagind db. For Replication Mode,
+    this server id must be greater than the source db server id.
+
+  - Staging DB Port
+
+    Port for the dsource (staging db).
+
+  - Staging Initialization Password
+
+    Password to use while initializing the dsource (staging db).
+    This password will be assigned to the *'root'@'localhost'*
+
+  - MySQL Base Directory
+
+    MySQL installation directory. This is where the location of */bin/mysql*
+
+  - Mount Location on Staging Host
+
+    This is the mount directory for Delphix on the staging host. 
+    This location should be unique and empty.
+
+  - Source DB UserName
+
+    Delphix db user on the source database. In Manual Ingestion mode, Delphix does not connect to the source db.
+    Delphix will be using this db user to manage the dsource and other time travel operations.
+    This user must be part of the source db backup that will be restored into the staging db.
+
+  - Source DB Password
+
+    Password for the source db user.
+
+8. On dSource Configuration screen, select the dataset group where the dSource will be placed and click *Next*.
+9. On the Data Management screen, select the staging environment and the environment user and click *Next*.
+10. On the Policies screen, select the Snapsync and Retention policies for the dSource and click *Next*.
+11. On the Hooks screen, add any pre-sync and (or) post-sync hooks as required and click *Next*.
+12. Review the dSource configuration on the Summary screen and Submit. The Linking process has commenced.
+
+    ![Screenshot](../../image/dsource-creation.png)
+
+
+Once the dSource creation is successful, the <span class="code_title">*Timeflow*</span> tab should show the initial snapshot.
+
+Done, What's Next?
+----------------
+Alright, we have created a MySQL dSource. Next step - Provision a VDB.   
+