3.1.2 (#13)

* Starting 3.1.1

* Add parse failure to oltp with external control

* errortime on also for debug RWL-601

* oltpforever new -s option; longoptions

* oltpforever debug version by mistake

* Don't drop session after banner

* Only set OCI_ATTR_CONNECTION_CLASS on session pool if explictly given

* Missing comment on cclass change

* Improve misc output

* Fix bug with ampersand at end of SQL

* Fix bug with ampersand at end of sql

* Improve some output resolution

* rwl_min/max_pool in env file

* gnuplotjs empty implies mousing standalone

* Fix some double issues with sprintf, rwld2s

* Arrival rate bursts in oltp can be repeated

* Change burst_repeat, fix running plots

* Plot dropped if time_out on oltprun command line

* Add burst_function

* Misc code for distributing oltpcore/mtitcore processes

* Reservere test/362.rwl for trig functions

* In 24h sim, make no hard parse/full scan the default

* Add key option to oltpplot, oltpdirectory

* Add oltprunnumber

* trig code changes and test

* sin cos atan2 added to rwlman

* Fix RWL-236 / mute141

* Add title to running throughput plot

* oltpplot --gnuplot-extra option

* Final trigonometry stuff

* $oraerror:nocount directive

* Even more possibilities to change graphs in oltpplot

* Typo fixes

* External control parsefailure updates

* oltpplot --no-client-cpu option

* oltpplot --no-x2label

* oltpplot: --ses-simplified option

* CHanged text with --ses-simplified

* xc logoff in more transactions

* Allow not printing parse failure to stderr

* ignore_parsefailure can be set ahead of oltpxcrun

* Some doc updates, new oltp tags file

* Cleanup bison warnings from missing %empty

* Fix a shift/reduce conflict

* Nicer html man pages using monospace for code

* Missing ref from statement to simpledeclaration

* Add ref to rwlprogram in statement rwlman page

* A few error message changes

* Update ERROR.md, ERRORLIST.md

* $oraerror:nocount is now default

* RWL-278 reported on wrong line

* Test for incorrect RWL-217

* Create global awrreport

* Add global awr generation

* Wrap long awr lists

* Add percentiles_oltp view for the oltp workload

* fflush to make tail nicer

* Add batch pool counts to cli.csv

* Add plot of total open sessions client side

* Change rampup for rwl_batch

* Use *rand48_r on Linux, move all includes to rwl.h

* Provide own rand48 implementation

* Minor changes ro rwlrand48

* Replace getopt_long with own rwlgetopt

* A few getopt fixes and code hardening

* Windows \r\n stuff plus a few minor things

* Updated ERRORLIST.md

* Include bison/flex generated files in repository

* Windows port

* readlob with length and offset

* Started writelob-lo, changes to relevant tests

* Always find public directory; windows updates

* oltpxc has cpu hog and wait

* Three digits in double xc_marker

* Windows slash convert and crnlreadline

* Some Windows stuff

* Cannot use regex in rwlbdident due to Windows

* Some windows changes

* Undo some incorrectly added obj files

* Even more Windows work

* Various MS Windows fixes

* Allow sql_id() for anonymous sql

* ociping uses filecommands.rwl

* Windows slash conversions

* ashplot on Windows

* Update generated files

* ashplot, ociping on Windows

* More Windows updates

* Fix typo

* No error if dir exists

* Windows documentation

* Fix rwlunixepoch on Windows

* Fix some timing etc on Windows

* Changes to make and ship on Windows

* $filelinename directive

* wrong ++

* Recreated most html

* Changes for dtime and atime

* Only plot atime, dtime if available

* Updated ERRORLIST.md

* ERRORLIST.md is generated

* OCIPing wrapped with o/d time

* log scale default for end-to-end time graphs

* HOMEDRIVE/HOMEPATH on Windows

* Few man changes

* include oci.h after all system includes

* Windows command and makefile

* Few doc updates before 3.1.1 release

* 3.1.1 release

* Add .cmd files

* proper name for .zip

* Doc updates for 3.1.1

* writelob

---------

Co-authored-by: Jack Kennedy <jack.f.kennedy@oracle.com>

Author: bengsig

.gitignore

@@ -5,11 +5,14 @@ new.git.branch
 .nfs*
 afiedt.buf
 bin/rwloadsim??
+bin/*.exe
 bin/rwloadsim.vg
 bin/rwlerror
 genbin11/*
 genbin19/*
 lib/*.o
+lib/*.obj
+lib/*.lib
 dist-rwl-bin.sh
 push_to_remotes.sh
 make.generated.sh

CHANGELOG.md

@@ -1,5 +1,18 @@
 # RWP\*Load Simulator Release Notes
 
+## 3.1.2
+
+* Bug fixes
+* Statisitcs count only increased when no ORA- error during execution
+* Add parsefailure to oltp external control
+* awrreport can generate global report 
+* You can use sql\_id() without a name to get sqlid of immediate or embedded sql
+* rwloadsim itself is available on Microsoft Windows; not the oltp workload
+* Changes to repository
+* Full end-to-end tracing (on Linux only)
+
+Note that there was no release 3.1.1
+
 ## 3.1.0
 
 * Bug fixes

admin/.vim/syntax/rwl.vim

@@ -1,7 +1,10 @@
 " Vim syntax file
 " Language: RWP*Worklod Simualator
 " Maintainer: Bjørn Kisbye Ensig
-" Latest Revision: 13 sep 2023
+" Copyright (c) 2023 Oracle Corporation
+" Licensed under the Universal Permissive License v 1.0
+" as shown at https://oss.oracle.com/licenses/upl/
+" Latest Revision: 28 feb 2024
 
 if exists("b:current_syntax")
   finish
@@ -23,7 +26,7 @@ syn keyword rwlKeyword getenv system opensessioncount activesessioncount serverr
 syn keyword rwlKeyword unsigned hexadecimal octal printf fprintf encode decode elseif
 syn keyword rwlKeyword connectionpool connectionclass sprintf global querynotification
 syn keyword rwlKeyword normalrandom statisticsonly ceil trunc floor 
-syn keyword rwlKeyword lobprefetch readlob writelob
+syn keyword rwlKeyword lobprefetch readlob writelob atan2 winslashf2b winslashf2bb
 syn keyword rwlKeyword getrusage instr instrb regexextract nextgroup=rwlNumber skipwhite
 
 syn match rwlVariable "\$#"
@@ -38,7 +41,7 @@ syn match rwlNumber '\<\d\+[Ee][+-]\?\d\+'
 syn match rwlNumber '\<\d\+\.\d*[Ee][+-]\?\d\+'
 
 syn match rwlComment "#.*$" 
-syn region rwlString	start=+"+ skip=+\\\n\\\\\|\\"+ end=+"+
+syn region rwlString	start=+"+ skip=+\\\n\|\\\\\|\\"+ end=+"+
 
 syn match rwlString '\$[A-Z][_A-Za-z0-9]*'
 
@@ -94,6 +97,10 @@ syn region rwlSQL	start='\<\ccreate[ \t\n]\+\(or[ \t\n]\+replace[ \t\n]\+\)\?lib
 syn match rwlDirective '\$statistics:basic'
 syn match rwlDirective '\$statistics:histograms'
 syn match rwlDirective '\$statistics:all'
+syn match rwlDirective '\$statsdbtime:on'
+syn match rwlDirective '\$statsdbtime:off'
+syn match rwlDirective '\$statsapptime:on'
+syn match rwlDirective '\$statsapptime:off'
 syn match rwlDirective '\$ampersand:on'
 syn match rwlDirective '\$ampersand:off'
 syn match rwlDirective '\$queueeverytiming:on'
@@ -113,6 +120,8 @@ syn match rwlDirective '\$reconnect1017:off'
 syn match rwlDirective '\$pre31fileassign:warn'
 syn match rwlDirective '\$pre31fileassign:on'
 syn match rwlDirective '\$pre31fileassign:off'
+syn match rwlDirective '\$trigonometry:radians'
+syn match rwlDirective '\$trigonometry:degrees'
 syn match rwlDirective '\$niceabort:on'
 syn match rwlDirective '\$niceabort:off'
 syn match rwlDirective '\$queue:on'
@@ -133,6 +142,7 @@ syn match rwlDirective '\$ora01013:stop'
 syn match rwlDirective '\$ora01013:continue'
 syn match rwlDirective '\$oraerror:stop'
 syn match rwlDirective '\$oraerror:continue'
+syn match rwlDirective '\$oraerror:nocount'
 syn match rwlDirective '\$oerstatistics:on'
 syn match rwlDirective '\$oerstatistics:off'
 syn match rwlDirective '\$implicit:off'
@@ -152,15 +162,16 @@ syn match rwlDirective '\$randseed:[0-9a-fA-F][0-9a-fA-F]*'
 syn match rwlDirective '\$randseed:0[xX][0-9a-fA-F][0-9a-fA-F]*'
 syn match rwlDirective '\$longoption:[a-z][-a-z]*'
 syn match rwlDirective '\$longoption:[a-z][-a-z]*=[^ ][^ ]*'
-syn match rwlDirective '\$userhelp:"[^"]*"'
+syn match rwlDirective '\$userhelp:"[^\n]*"'
 syn match rwlDirective '\$useroption:[a-zA-Z][-0-9a-zA-Z_]*'
-syn match rwlDirective '\$useroption:[a-zA-Z][-0-9a-zA-Z_]*:"[^"]*"'
+syn match rwlDirective '\$useroption:[a-zA-Z][-0-9a-zA-Z_]*:"[^\n]*"'
 syn match rwlDirective '\$userswitch:[a-zA-Z][-0-9a-zA-Z_]*'
-syn match rwlDirective '\$userswitch:[a-zA-Z][-0-9a-zA-Z_]*:"[^"]*"'
+syn match rwlDirective '\$userswitch:[a-zA-Z][-0-9a-zA-Z_]*:"[^\n]*"'
 syn match rwlDirective '\$debugon:[0-9A-Za-z][0-9a-zA-Z,]*'
 syn match rwlDirective '\$debugoff:[0-9A-Za-z][0-9a-zA-Z,]*'
 syn match rwlDirective '\$include:"[^"<>]*"'
 syn match rwlDirective '\$include:<[^"<>]*>'
+syn match rwlDirective '\$filelinename:[0-9][0-9]*:"[^"<>]*"'
 syn match rwlDirective '\$sqllogging:file:"[^"<>]*"'
 syn match rwlDirective '\$sqllogging:append:"[^"<>]*"'
 syn match rwlDirective '\$sqllogging:stdout'
@@ -170,13 +181,27 @@ syn match rwlDirective '\$if[^o]'
 syn match rwlDirective '\$then'
 syn match rwlDirective '\$else'
 syn match rwlDirective '\$endif'
+syn match rwlDirective '\$linux'
+syn match rwlDirective '\$solaris'
+syn match rwlDirective '\$windows'
+syn match rwlDirective '\$macos'
 syn match rwlDirective '\$embeddeddmlarray:\d\+' 
 syn match rwlDirective '\$embeddedqueryarray:\d\+' 
 syn match rwlDirective '\$bindoutname:on'
 syn match rwlDirective '\$bindoutname:on:[a-zA-Z][0-9a-zA-Z_]*'
 syn match rwlDirective '\$bindoutname:off'
 syn match rwlDirective '\$musymbol:[^ ][^ ]*'
 syn match rwlDirective '\$hostname:[a-zA-Z0-9_][-a-zA-Z0-9_\.]*'
+syn match rwlDirective '\$slashconvert:on'
+syn match rwlDirective '\$slashconvert:off'
+syn match rwlDirective '\$crnlreadline:on'
+syn match rwlDirective '\$crnlreadline:off'
+syn match rwlDirective '\$crnlwriteline:on'
+syn match rwlDirective '\$crnlwriteline:off'
+syn match rwlDirective '\$crnlstring:on'
+syn match rwlDirective '\$crnlstring:off'
+syn match rwlDirective '\$crnlgeneral:on'
+syn match rwlDirective '\$crnlgeneral:off'
 
 
 let b:current_syntax = "rwl"

admin/rwl312.sql

@@ -0,0 +1,47 @@
+-- update the RWP*Load Simulator repository
+-- for version 3.1.1
+--
+-- Copyright (c) 2023 Oracle Corporation
+-- Licensed under the Universal Permissive License v 1.0
+-- as shown at https://oss.oracle.com/licenses/upl/
+--
+-- Changes
+-- 
+-- NAME     DATE         COMMENTS
+-- bengsig  19-dec-2023  Add ignore_parsefailure column to oltpxc
+-- bengsig   3-oct-2023  Add parsefailure column to oltpxc
+-- bengsig  25-apr-2023  Creation
+--
+alter table oltpxc add
+( parsefailure number 
+, ignore_parsefailure number)
+/
+
+update oltpxc set parsefailure = 0
+, ignore_parsefailure=0
+/
+commit
+/
+
+alter table oltpxc add
+( cpusec number
+, waitsec number
+)
+/
+update oltpxc set cpusec=0
+, waitsec=0
+/
+commit
+/
+
+alter table persec add
+( atime number (*,6)
+, dtime number (*,6) 
+)
+/
+
+alter table runres add
+( atime number (*,6) 
+, dtime number (*,6) 
+)
+/

admin/rwloadsim.sql

@@ -7,6 +7,7 @@
 -- Changes
 -- 
 -- NAME     DATE         COMMENTS
+-- bengsig   1-mar-2024  atime,dtime column
 -- bengsig  25-apr-2022  MTIT
 -- bengsig   1-nov-2022  Make sequence nocache to prevent many lost numbers
 -- bengsig  12-oct-2022  Add wtime,etime to persec
@@ -44,6 +45,8 @@ create table runres
 , vname     varchar2(30) not null
 , wtime     number(*,6)
 , etime     number(*,6)
+, atime     number(*,6)
+, dtime     number(*,6)
 , ecount    number
 , tcount    number
 , constraint runres_pk primary key(runnumber, procno, vname)
@@ -70,6 +73,8 @@ create table persec
 , scount    number
 , wtime     number(*,6)
 , etime     number(*,6)
+, atime     number(*,6)
+, dtime     number(*,6)
 , constraint persec_pk primary key(runnumber, procno, vname, second)
 )
 organization index
@@ -181,6 +186,9 @@ create table oltpxc
 , affinity number
 , rowspins number
 , arraysize number
+, parsefailure number
+, cpusec number
+, waitsec number
 , constraint oltpxc_pk primary key(key, hostname)
 )
 /

admin/rwlviews.sql

@@ -5,6 +5,8 @@
 -- as shown at https://oss.oracle.com/licenses/upl/
 
 -- History
+-- bengsig   1-mar-2024 - dtime, atime
+-- bengsig  23-jan-2024 - Add percentiles_oltp for use by the oltp workload
 -- bengsig  12-oct-2022 - Updated persec_a with wtime,etime
 -- bengsig  09-sep-2020 - Remove legacy
 -- bengsig         2017 - Creation
@@ -44,6 +46,8 @@ select
 , sum(scount) scount
 , sum(wtime)  wtime
 , sum(etime)  etime
+, sum(atime)  atime
+, sum(dtime)  dtime
 from persec
 group by runnumber, vname, second
 /
@@ -54,9 +58,13 @@ create or replace view runres_a
 , vname
 , wtime
 , etime
+, atime
+, dtime
 , tcount
 , avgw
 , avge
+, avga
+, avgd
 , ecount
 )
 -- aggregate multi process runs
@@ -67,9 +75,13 @@ select
 , vname
 , sum(wtime) wtime
 , sum(etime) etime
+, sum(atime) atime
+, sum(dtime) dtime
 , sum(tcount) tcount
 , avg(decode(ecount,0,null,wtime/ecount)) avgw
 , avg(decode(ecount,0,null,etime/ecount)) avge
+, avg(decode(ecount,0,null,atime/ecount)) avga
+, avg(decode(ecount,0,null,dtime/ecount)) avgd
 , sum(ecount) ecount
 from runres
 group by runnumber, vname
@@ -211,3 +223,124 @@ group by runnumber
 -- ... and do the group by on the rounded value
 , round(second)
 /
+
+-- The following view is used by the oltp workload
+-- and takes skipped transactions into account
+create or replace view percentiles_oltp
+-- show selected total execution time percentiles
+as
+select x.runnumber
+, x.vname
+, avg(y.avge+y.avgw/*+nvl(y.avgq,0)*/) avgt
+, avg(pct50) pct50
+, avg(pct90) pct90
+, avg(pct95) pct95
+, avg(pct98) pct98
+, avg(pct99) pct99
+, avg(pct995) pct995
+, avg(pct998) pct998
+, avg(pct999) pct999
+, avg(pct9995) pct9995
+from
+(
+select
+runnumber
+, vname
+, case when x5000 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x5000-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct50
+, case when x9000 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9000-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct90
+, case when x9500 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9500-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct95
+, case when x9800 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9800-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct98
+, case when x9900 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9900-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct99
+, case when x9950 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9950-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct995
+, case when x9980 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9980-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct998
+, case when x9990 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9990-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct999
+, case when x9995 not between leftt and rightt then null else
+  leftb+(rightb-leftb)*(x9995-leftt)
+  /decode(rightt,leftt,null,rightt-leftt)
+  end pct9995
+from
+(
+select runnumber
+, vname
+--, buckno
+, bucktim/2 leftb
+, bucktim rightb
+, bcount
+, nvl(sum(bcount)
+      over
+      ( partition by runnumber, vname
+        order by bucktim
+	rows between unbounded preceding
+	and 1 preceding)
+     , 0) leftt
+, sum(bcount) over (partition by runnumber, vname order by bucktim) rightt
+, sum(bcount) over (partition by runnumber, vname) gtotal
+, sum(bcount) over (partition by runnumber, vname) * 0.5    x5000
+, sum(bcount) over (partition by runnumber, vname) * 0.9    x9000
+, sum(bcount) over (partition by runnumber, vname) * 0.95   x9500
+, sum(bcount) over (partition by runnumber, vname) * 0.98   x9800
+, sum(bcount) over (partition by runnumber, vname) * 0.99   x9900
+, sum(bcount) over (partition by runnumber, vname) * 0.995  x9950
+, sum(bcount) over (partition by runnumber, vname) * 0.998  x9980
+, sum(bcount) over (partition by runnumber, vname) * 0.999  x9990
+, sum(bcount) over (partition by runnumber, vname) * 0.9995  x9995
+, 100 * sum(bcount) over (partition by runnumber, vname order by bucktim)
+  / sum(bcount) over (partition by runnumber, vname)
+  bcpct
+from ( select runnumber, vname, buckno, bucktim, bcount from histogram_a
+  union all
+  -- the following will add an artificial row with above all real rows
+  -- which distributes the call_failure rows onto the vnames
+  -- by the distrubution of really done calls
+  select good.runnumber
+  , good.vname
+  , 30 buckno -- This value MUST match #define RWL_MAX_HIST_BUCK 30 in rwl.h
+  , 2048 bucktim -- and the same here
+  , fail.failcount * good.ecount / decode(fail.totalcount,0,null,fail.totalcount) bcount
+  from
+  (
+  select runnumber, vname, ecount from runres_a
+  where vname != 'call_failure'
+  ) good
+  left join
+  (
+  select runnumber
+  , sum(decode(vname,'call_failure',ecount,0)) failcount
+  , sum(decode(vname,'call_failure',0,ecount)) totalcount
+  from runres_a
+  group by runnumber
+  ) fail
+  on good.runnumber = fail.runnumber
+)
+)
+) x
+join runres_a y
+on x.runnumber = y.runnumber
+and x.vname = y.vname
+group by x.runnumber, x.vname
+/
+