Initial commit

Author: mgoddard

.gitignore

@@ -0,0 +1 @@
+gen_data

README.md

@@ -0,0 +1,32 @@
+# Stock Market Data Generator for CockroachDB
+
+This is a simple Go program to generate and load into CockroachDB some stock market oriented data.
+It was built in response to a question about how many rows of this type of data could be loaded
+within a specified time window.
+
+## Build
+```
+$ go get github.com/google/uuid
+$ go get github.com/jackc/pgx/v4
+$ go build
+```
+
+## Run
+
+* Edit the [environment file](./env.sh), setting the various DB connection parameters as well
+  as:
+  - `BATCH_SIZE`: the number of rows copied into the table at a time (defaults to 128)
+  - `N_THREADS`: the number of parallel goroutines used to parallelize the work (defaults to 4).
+    NOTE: each of these consumes one DB connection
+
+* Dump the table DDL and use the output to create the table to hold this generated data:
+  `$ ./gen_data --dump-ddl`
+
+* Generate and load the data:
+  `$ ./gen_data 100000`
+
+## Notes
+
+* The [data](./data) directory contains some data files used as reference here since it seemed like
+"stock market data".
+

data/prices.csv

@@ -0,0 +1,78 @@
+symbol,end_date,amend,period_focus,fiscal_year,doc_type,revenues,op_income,net_income,eps_basic,eps_diluted,dividend,assets,cur_assets,cur_liab,cash,equity,cash_flow_op,cash_flow_inv,cash_flow_fin
+AMS,2016-12-31,False,FY,2016,10-K,18700000.0,3286000.0,930000.0,0.17,0.17,0.0,60598000.0,8388000.0,8681000.0,2871000.0,27173000.0,8378000.0,-1612000.0,-6104000.0
+APHB,2016-12-31,False,FY,2016,10-K,260000.0,-23378000.0,-24276000.0,-2.47,-2.47,0.0,18195000.0,6355000.0,3580000.0,5711000.0,9723000.0,-10589000.0,-279000.0,7209000.0
+CKX,2016-12-31,False,FY,2016,10-K,863402.0,187360.0,170674.0,0.09,0.09,0.0,14418324.0,4537058.0,132457.0,1081188.0,13986948.0,280807.0,-1973550.0,6507.0
+GSB,2016-12-31,False,FY,2016,10-K,33336000.0,5818000.0,3951000.0,0.19,0.18,0.06,50180000.0,19303000.0,16367000.0,8895000.0,29876000.0,7066000.0,-13880000.0,-176000.0
+HMG,2016-12-31,False,FY,2016,10-K,66030.0,-1405535.0,-431427.0,-0.42,-0.42,0.5,22777421.0,,,3019463.0,20397600.0,-880011.0,1687451.0,-9001362.0
+INTT,2016-12-31,False,FY,2016,10-K,40227000.0,4146000.0,2658000.0,0.26,0.26,0.0,42844000.0,38006000.0,5056000.0,28611000.0,37788000.0,4288000.0,-339000.0,-978000.0
+MHH,2016-12-31,False,FY,2016,10-K,132008000.0,4507000.0,2520000.0,0.57,0.56,0.0,39491000.0,22964000.0,12257000.0,829000.0,19072000.0,2292000.0,-38000.0,-2273000.0
+VSR,2016-07-01,False,FY,2016,10-K,167917000.0,-35937000.0,-37887000.0,-3.84,-3.84,0.0,61316000.0,51965000.0,54199000.0,1549000.0,1068000.0,-2072000.0,-11496000.0,13009000.0
+IOTS,2016-12-31,False,FY,2016,10-K,43968000.0,-10303000.0,-11612000.0,-0.77,-0.77,0.0,46183000.0,31579000.0,15408000.0,19719000.0,16365000.0,-5039000.0,-2481000.0,4221000.0
+ALBO,2016-12-31,False,FY,2016,10-K,11364000.0,-12294000.0,-16350000.0,-13.19,-13.19,0.0,49660000.0,30861000.0,12708000.0,29931000.0,36952000.0,-8784000.0,25480000.0,7612000.0
+LPTN,2016-12-31,False,FY,2016,10-K,64868000.0,-20761000.0,-41167000.0,-105.69,-105.69,0.0,102124000.0,43621000.0,20280000.0,19111000.0,42417000.0,-12901000.0,-2165000.0,12617000.0
+AAME,2016-12-31,False,FY,2016,10-K,166077000.0,929000.0,2237000.0,0.11,0.11,0.0,318600000.0,,,13252000.0,105506000.0,1316000.0,-2232000.0,-1454000.0
+BIOC,2016-12-31,False,FY,2016,10-K,3223096.0,-18025037.0,-18399322.0,-1.92,-1.92,0.0,7578326.0,5771995.0,4393552.0,4609332.0,658661.0,-15697058.0,-451403.0,11936464.0
+BYFC,2016-12-31,False,FY,2016,10-K,16334000.0,,3477000.0,0.12,0.12,0.0,429083000.0,,,1516000.0,45526000.0,930000.0,-74104000.0,23765000.0
+CALM,2017-02-25,False,Q3,2017,10-Q,306540000.0,-4573000.0,4139000.0,0.09,0.09,0.0,1085275000.0,484801000.0,92577000.0,31905000.0,868362000.0,-57217000.0,66562000.0,-6486000.0
+CART,2016-12-31,False,FY,2016,10-K,125000.0,,1119000.0,0.24,0.24,0.0,374917000.0,,,26149000.0,29033000.0,-29000.0,-8800000.0,39366000.0
+CLSN,2016-12-31,False,FY,2016,10-K,2500000.0,-20649820.0,-22053726.0,-0.85,-0.85,0.0,30849691.0,4512578.0,8423287.0,2624162.0,6725826.0,-18379364.0,9060713.0,2677669.0
+CYTX,2016-12-31,False,FY,2016,10-K,4656000.0,-19706000.0,-22046000.0,-1.28,-1.28,0.0,34609000.0,18747000.0,12501000.0,12560000.0,10986000.0,-19533000.0,64000.0,17609000.0
+DNBF,2016-12-31,False,FY,2016,10-K,5737000.0,,4979000.0,1.56,1.55,0.28,1070685000.0,,,22103000.0,94840000.0,10510000.0,3376000.0,-12902000.0
+EBIO,2016-12-31,False,FY,2016,10-K,29981000.0,2866000.0,1891000.0,0.09,0.09,0.0,104097000.0,25927000.0,3980000.0,25342000.0,38677000.0,2622000.0,461000.0,-13820000.0
+EMCF,2016-12-31,False,FY,2016,10-K,27080000.0,,3986000.0,1.86,1.85,1.04,692135000.0,,,2758000.0,54073000.0,5660000.0,-14513000.0,14875000.0
+FBSS,2016-12-31,False,FY,2016,10-K,26871000.0,,3674000.0,0.98,0.98,0.48,624445000.0,,,67846000.0,54451000.0,6574000.0,-11877000.0,19934000.0
+FNJN,2016-12-31,False,FY,2016,10-K,18387000.0,350000.0,-6439000.0,-0.28,-0.28,0.0,18305000.0,15036000.0,3814000.0,13678000.0,886000.0,1328000.0,-559000.0,6808000.0
+GPIC,2016-12-31,False,FY,2016,10-K,82139000.0,7560000.0,5183000.0,0.65,0.64,0.12,79999000.0,39092000.0,14741000.0,10604000.0,57533000.0,3547000.0,-8319000.0,-2281000.0
+GTXI,2016-12-31,False,FY,2016,10-K,,-25933000.0,-17724000.0,-1.22,-1.22,0.0,24502000.0,24298000.0,4611000.0,8910000.0,19891000.0,-20778000.0,2151000.0,13481000.0
+GTYHU,2016-12-31,False,FY,2016,10-K,,,,,,0.0,553504760.0,1240986.0,88739.0,1219822.0,5000001.0,-85873.0,-552000000.0,
+GFED,2016-12-31,False,FY,2016,10-K,50332.0,,5594011.0,1.28,1.27,0.34,687979819.0,,,9088441.0,69974380.0,8469338.0,-48033060.0,29877744.0
+HWCC,2016-12-31,False,FY,2016,10-K,261644000.0,-6535000.0,-6006000.0,-0.37,-0.37,0.15,175870000.0,126978000.0,24835000.0,,90131000.0,17243000.0,-33684000.0,16441000.0
+ICAD,2016-12-31,False,FY,2016,10-K,26338000.0,-9970000.0,-10099000.0,-0.63,-0.63,0.0,38651000.0,19933000.0,12855000.0,8585000.0,25038000.0,-5478000.0,-355000.0,-862000.0
+INVE,2016-12-31,False,FY,2016,10-K,56168000.0,-11220000.0,-13695000.0,-1.25,-1.25,0.0,40600000.0,31652000.0,22566000.0,9116000.0,3933000.0,-6217000.0,-549000.0,-221000.0
+ISNS,2016-12-31,False,FY,2016,10-K,14142000.0,687000.0,687000.0,0.14,0.14,0.0,8204000.0,4980000.0,1995000.0,1547000.0,6209000.0,447000.0,-1418000.0,-17000.0
+INTX,2016-12-31,False,FY,2016,10-K,175662000.0,-27599000.0,-30469000.0,-1.31,-1.31,0.0,52857000.0,31411000.0,29192000.0,10857000.0,7367000.0,-3921000.0,-6609000.0,10237000.0
+JSYN,2016-12-31,False,FY,2016,10-K,,-541802.0,-434228.0,-0.23,-0.23,0.0,40481763.0,8341.0,,1438.0,5000015.0,-371942.0,-40365000.0,40702055.0
+JTPY,2016-12-31,False,FY,2016,10-K,56330000.0,-8088000.0,-14593000.0,-0.89,-0.89,0.0,187210000.0,109633000.0,104615000.0,12584000.0,10209000.0,1530000.0,-1623000.0,7083000.0
+NBRV,2016-12-31,False,FY,2016,10-K,6482000.0,-55047000.0,-54890000.0,-25.56,-25.56,0.0,93240000.0,90621000.0,15877000.0,32778000.0,77256000.0,-49321000.0,23352000.0,22301000.0
+OVLY,2016-12-31,False,FY,2016,10-K,36702000.0,,7665000.0,0.95,0.95,0.0,1002110000.0,,,179025000.0,82450000.0,1810000.0,-107161000.0,-1940000.0
+OPGN,2016-12-31,False,FY,2016,10-K,4025685.0,-19009337.0,-19499303.0,-1.1,-1.1,0.0,8984045.0,5681758.0,5271937.0,4117324.0,3167481.0,-17250637.0,-123514.0,13664690.0
+ANF,2017-01-28,False,FY,2016,10-K,3326740000.0,15188000.0,3956000.0,0.06,0.06,0.8,2295757000.0,1139300000.0,486000000.0,547189000.0,1252039000.0,184591000.0,-136746000.0,-83793000.0
+PESI,2016-12-31,False,FY,2016,10-K,51219000.0,-15792000.0,-13405000.0,-1.15,-1.15,0.0,65335000.0,14607000.0,16738000.0,163000.0,30871000.0,104000.0,-415000.0,-956000.0
+PZRX,2016-12-31,False,FY,2016,10-K,0.0,-18330000.0,-20137000.0,-2.68,-2.68,0.0,16448000.0,16177000.0,2024000.0,9983000.0,9297000.0,-9811000.0,-5664000.0,22168000.0
+AMID,2016-12-31,False,FY,2016,10-K,160950000.0,-23268000.0,-3470000.0,-1.11,,0.0,1563495000.0,48731000.0,77482000.0,2939000.0,,45362000.0,-551441000.0,509018000.0
+PBSK,2016-12-31,False,FY,2016,10-K,22127000.0,,1777000.0,0.5,0.5,0.28,458465000.0,,,24389000.0,68701000.0,5550000.0,-25793000.0,20756000.0
+WAAS,2016-12-31,False,FY,2016,10-K,114100000.0,-12195000.0,-20472000.0,-0.28,-0.28,0.0,536696000.0,123620000.0,47738000.0,95334000.0,367506000.0,13557000.0,-65093000.0,129068000.0
+BBY,2017-01-28,False,FY,2017,10-K,39403000000.0,1854000000.0,1228000000.0,3.86,3.81,1.57,13856000000.0,10516000000.0,7122000000.0,2240000000.0,4709000000.0,2545000000.0,-887000000.0,-1404000000.0
+ROST,2017-01-28,False,FY,2016,10-K,12866757000.0,,1117654000.0,2.85,2.83,0.54,5309351000.0,2813049000.0,1752506000.0,1111599000.0,2748017000.0,1558851000.0,-292763000.0,-916091000.0
+BOX,2017-01-31,False,FY,2017,10-K,398605000.0,-150655000.0,-151787000.0,-1.19,-1.19,0.0,493674000.0,322101000.0,297941000.0,177391000.0,74732000.0,-1218000.0,-7572000.0,479000.0
+SCHL,2017-02-28,False,Q3,2017,10-Q,336200000.0,-23600000.0,-15400000.0,-0.44,-0.44,0.15,1847600000.0,1054600000.0,508100000.0,461800000.0,1269000000.0,113400000.0,-45600000.0,-5500000.0
+SCKT,2016-12-31,False,FY,2016,10-K,20787588.0,2563016.0,12147088.0,2.1,1.8,0.0,20638995.0,5982786.0,4116375.0,1319006.0,16169932.0,879816.0,-304470.0,-194495.0
+SNGX,2016-12-31,False,FY,2016,10-K,10448794.0,-5709582.0,-3245383.0,-0.93,-1.34,0.0,10267105.0,10113775.0,2869857.0,8772567.0,7397248.0,-4982421.0,-7159.0,8840602.0
+SPHS,2016-12-31,False,FY,2016,10-K,,,-11164000.0,-0.49,-0.49,0.0,29998000.0,29975000.0,2221000.0,12800000.0,14324000.0,-10329000.0,-13721000.0,30971000.0
+SRNE,2016-12-31,True,FY,2016,10-K,8152000.0,-96777000.0,-60923000.0,-1.21,-1.21,0.0,401586000.0,89654000.0,76085000.0,82398000.0,86502000.0,-70928000.0,-17452000.0,131740000.0
+SFBC,2016-12-31,False,FY,2016,10-K,30167000.0,,5378000.0,2.16,2.09,0.3,588383000.0,,,54582000.0,60275000.0,7148000.0,-42898000.0,42068000.0
+SPWH,2017-01-28,False,FY,2016,10-K,779956000.0,60687000.0,29669000.0,0.7,0.7,0.0,346248000.0,255924000.0,147219000.0,1911000.0,30001000.0,15482000.0,-27494000.0,11814000.0
+SBCP,2016-12-31,False,FY,2016,10-K,104000.0,,-43000.0,-0.01,-0.01,0.0,931435000.0,,,50273000.0,112101000.0,-2615000.0,-99645000.0,93189000.0
+TLF,2016-12-31,False,FY,2016,10-K,82923992.0,10300731.0,6402259.0,0.69,0.69,0.0,70652720.0,53314242.0,8246068.0,16862304.0,53693201.0,7539583.0,-1545348.0,-94545.0
+THLD,2016-12-31,False,FY,2016,10-K,0.0,-24362000.0,-24094000.0,-0.34,-0.34,0.0,24283000.0,24174000.0,2616000.0,10551000.0,19888000.0,-25099000.0,26033000.0,28000.0
+DKS,2017-01-28,False,FY,2016,10-K,7921981000.0,449854000.0,287396000.0,2.59,2.56,0.605,4058296000.0,1995678000.0,1397415000.0,164777000.0,1929489000.0,758983000.0,-550324000.0,-162865000.0
+DDS,2017-01-28,False,FY,2016,10-K,6256971000.0,,169220000.0,4.93,4.93,0.28,3888136000.0,1837921000.0,976517000.0,346985000.0,1717417000.0,517007000.0,-119649000.0,-253242000.0
+DG,2017-02-03,False,FY,2016,10-K,21986598000.0,2063449000.0,1251133000.0,4.45,4.43,1.0,11672298000.0,3677771000.0,2622805000.0,187915000.0,5406294000.0,1605041000.0,-550936000.0,-1024137000.0
+TRCB,2016-12-31,False,FY,2016,10-K,40113000.0,,8631000.0,1.04,1.01,0.14,940211000.0,,,19844000.0,100716000.0,8906000.0,-79701000.0,66145000.0
+ENV,2016-12-31,False,FY,2016,10-K,578164000.0,-23444000.0,-55567000.0,-1.3,-1.3,0.0,872401000.0,113084000.0,155954000.0,52592000.0,413287000.0,76815000.0,-57927000.0,-18014000.0
+EXPR,2017-01-28,False,FY,2016,10-K,2192547000.0,103601000.0,57417000.0,0.73,0.73,0.0,1185189000.0,514133000.0,282397000.0,207373000.0,635687000.0,186708000.0,-108866000.0,-58271000.0
+WCFB,2016-12-31,False,FY,2016,10-K,4379572.0,,109413.0,0.05,0.05,0.16,123647347.0,,,1716578.0,28846980.0,640280.0,-17532456.0,11048193.0
+WEBK,2016-12-31,False,FY,2016,10-K,26514000.0,,2936000.0,1.26,1.24,0.15,695283000.0,,,3607000.0,55214000.0,3310000.0,-73229000.0,70166000.0
+WSCI,2017-02-26,False,Q2,2017,10-Q,6313586.0,,-362193.0,-0.12,-0.12,0.0,24603065.0,11603209.0,4104841.0,5648070.0,13357064.0,1436685.0,-76499.0,548560.0
+ZAIS,2016-12-31,False,FY,2016,10-K,31677000.0,,-5160000.0,-0.37,-0.37,0.0,514145000.0,,,38712000.0,89965000.0,-18148000.0,8402000.0,4430000.0
+GME,2017-01-28,False,FY,2016,10-K,8607900000.0,557700000.0,353200000.0,3.42,3.4,1.48,4975900000.0,2140700000.0,1761500000.0,669400000.0,2254100000.0,537100000.0,-578000000.0,238700000.0
+ZYNE,2016-12-31,False,FY,2016,10-K,7250.0,-23207628.0,-23388507.0,-2.58,-2.58,0.0,36554274.0,36410692.0,6966966.0,30965791.0,29587308.0,-20438166.0,-109628.0,10000525.0
+GES,2017-01-28,False,FY,2017,10-K,2209368000.0,22708000.0,22234000.0,0.27,0.27,0.9,1534485000.0,1044012000.0,345453000.0,396129000.0,980994000.0,71456000.0,-49993000.0,-68750000.0
+JCP,2017-01-28,False,FY,2016,10-K,12547000000.0,395000000.0,1000000.0,0.0,0.0,0.0,9314000000.0,4097000000.0,2419000000.0,887000000.0,1354000000.0,334000000.0,-316000000.0,-31000000.0
+NMBL,2017-01-31,False,FY,2017,10-K,402597000.0,-156617000.0,-158315000.0,-1.85,-1.85,0.0,333020000.0,280043000.0,145767000.0,184814000.0,107030000.0,-15596000.0,-24872000.0,14228000.0
+ORN,2016-12-31,False,FY,2016,10-K,602537000.0,4074000.0,-3620000.0,-0.13,-0.13,0.0,447676000.0,195108000.0,117520000.0,305000.0,226204000.0,23149000.0,-17686000.0,-6503000.0
+OXM,2017-01-28,False,FY,2016,10-K,1022588000.0,89884000.0,52461000.0,3.18,3.15,1.08,685159000.0,231628000.0,131396000.0,6332000.0,376130000.0,118565000.0,-146491000.0,27367000.0
+PVH,2017-01-29,False,FY,2016,10-K,8203100000.0,789200000.0,549000000.0,6.84,6.79,0.0375,11067900000.0,2879600000.0,1564800000.0,730100000.0,4804500000.0,954800000.0,-480900000.0,-303200000.0
+REX,2017-01-31,False,FY,2016,10-K,453799000.0,,32333000.0,4.91,4.91,0.0,454024000.0,226517000.0,22519000.0,188576000.0,388274000.0,69109000.0,-7747000.0,-8551000.0
+SCNB,2016-12-31,False,FY,2016,10-K,86202000.0,,19831000.0,1.67,1.66,0.4,2092291000.0,,,124854000.0,215028000.0,19822000.0,101770000.0,-94824000.0
+TLRD,2017-01-28,False,FY,2016,10-K,3378703000.0,132826000.0,24928000.0,0.51,0.51,0.72,2097872000.0,1165717000.0,459920000.0,70889000.0,-107618000.0,242628000.0,-99077000.0,-98777000.0