Hectares BC Performance

Performance Tests Description

Performance testing is somewhat difficult to measure when all the major elements are driven by user actions. A user may take a long time to perform the multi-step sequence required to retrieve his final result, and in the meantime many other queries will have been run before he has had a chance to finish.

We decided to time how long an average action would take to be performed using the User Interface(UI). Then we wrote scripts to simulate those actions (without the UI delays). What we are trying to find is how much processing time is required for a user action that takes X amount of time.

Additionally, these tests were run with a single user, two users (as per the requirements), and ten users.

Summary (How much...) Tab Performance

For the summary tab performance tests, the following sequence was performed: - Navigating to the summary tab - Opening a few layers in the tree - Dragging a few category items to the row/column headers - Dragging a couple of value items to the value drop area - Executing the query The above sequence took about 1 minute to perform via the user interface. Each iteration of all these steps is considered a single run and is referred to as such in the results tables. Here are the times taken by a script:

Number of users Number of runs Database connections Total time for 1 user Average time per run
1 10 10 35.568 3.5568
1 100 10 404.642 4.04642
2 100 10 362.573 3.62573
2 100 10 383.373 3.83373
10 10 2 139.07 13.907
10 10 2 173.342 17.3342
10 10 2 172.441 17.2441
10 10 2 170.471 17.0471
10 10 2 156.663 15.6663
10 10 2 138.578 13.8578
10 10 2 178.663 17.8663
10 10 2 189.064 18.9064
10 10 2 169.312 16.9312
10 10 2 134.844 13.4844
10 10 10 85.135 8.5135
10 10 10 97.234 9.7234
10 10 10 96.212 9.6212
10 10 10 93.915 9.3915
10 10 10 81.601 8.1601
10 10 10 71.938 7.1938
10 10 10 90.505 9.0505
10 10 10 80.202 8.0202
10 10 10 95.843 9.5843
10 10 10 89.923 8.9923

Referring to the above table, there are a few variables we are able to change to alter performance. One of which is the number of database connections. The number of DB connections should typically be greater than the number of users we intend to serve at any given time. However, it is also a good idea to limit that number if it too high and just allow the user requests to get queued up rather than overloading the DB with many numerous requests. The code has also been optimized to only require a single connection per user at any given time.

As for results, all the results with 10 DB connections are acceptable.

Land Characterization (Show me where...) Tab Performance

For the land characterization tab performance tests, the following sequence was performed: - Navigating to the land characterization tab - Opening a few layers in the tree - Dragging a few tree items into the definition box - Executing the query The above sequence took about 1 minute to perform via the user interface. Each iteration of all these steps is considered a single run and is referred to as such in the results tables. Here are the times taken by a script:

Number of users Number of runs Database connections Total time for 1 user Average time per run
1 10 10 7.83 0.783
1 100 10 74.581 0.74581
2 100 10 74.981 0.74981
2 100 10 69.752 0.69752
10 10 2 16.146 1.6146
10 10 2 18.804 1.8804
10 10 2 20.74 2.074
10 10 2 17.54 1.754
10 10 2 14.735 1.4735
10 10 2 13.099 1.3099
10 10 2 17.02 1.702
10 10 2 17.23 1.723
10 10 2 16.382 1.6382
10 10 2 16.904 1.6904
10 10 10 16.062 1.6062
10 10 10 19.303 1.9303
10 10 10 16.722 1.6722
10 10 10 18.724 1.8724
10 10 10 16.838 1.6838
10 10 10 16.95 1.695
10 10 10 15.808 1.5808
10 10 10 16.31 1.631
10 10 10 14.339 1.4339
10 10 10 15.299 1.5299

Load Tests Description

The purpose of these tests was to determine the scalability of the master grids mechanisms and to see where the upper limit is for the number of data layers.

The first set of tests time how long it takes to reload the complete data set. This includes loading the full master grid table, creating indices, and subsampling to create the lower resolution master grid tables. The current set of data layers was used as a first timing point, as it provides a good distribution of data types and sizes. For other timing points, we modified the data loader to allow loading multiple copies of the original data. In conjunction, the metadata loader was modified to load multiple copies of the original metadata. This allowed empirical testing of the user interface's responsiveness.

Loading Performance

The following times were produced:

Number of Grids Run 1 Run 2 Run 3 Average Time
79 ~4 hours 4.85 hours 4.99 hours 4.61 hours
237 45105.953 seconds = 12.529 hours 45374.247 seconds = 12.604 hours 45143.363 seconds = 12.540 hours 12.558 hours
486 94396.533 seconds = 26.221 hours - - 26.221 hours

Examining the above table, loading three times the amount of data requires a little less than three times the amount of time. Similar results are seen with 6 times the amount of data. For the user interface, testing showed only minor impact on responsiveness. This impact is largely due to the increased number of elements to be handled by the user's web browser.

Conclusion

The results for both summary and land characterization types of runs do not suffer great degradation when faced with 10 simultaneous users. The results for data loading show a fairly direct and linear relationship to the amount of data. The user interface shows no significant impact due to an increased amount of data.





Archived Results

For archival purposes, the results from previous performance testing are archived here.

Summary (How much...) Tab Performance

Number of users Number of runs Database connections Total time for 1 user Average time per run
1 10 10 34.34 3.43
1 100 10 269.8 2.7
2 100 10 221.09 2.21
2 100 10 282.22 2.82
10 10 2 84.5 8.45
10 10 2 88.28 8.83
10 10 2 90.56 9.06
10 10 2 102.84 10.28
10 10 2 106.69 10.67
10 10 2 122.13 12.21
10 10 2 124.13 12.41
10 10 2 125.58 12.56
10 10 2 126.48 12.65
10 10 2 134.73 13.47
10 10 10 39.95 4
10 10 10 43.06 4.31
10 10 10 44.55 4.45
10 10 10 46.34 4.63
10 10 10 49.05 4.9
10 10 10 49.58 4.96
10 10 10 50.66 5.07
10 10 10 51.05 5.1
10 10 10 53.73 5.37
10 10 10 56.55 5.65

Land Characterization (Show me where...) Tab Performance

Number of users Number of runs Database connections Total time for 1 user Average time per run
1 10 10 8.42 0.84
1 100 10 80.11 0.8
2 100 10 99 0.99
2 100 10 108.38 1.08
10 10 2 54.47 5.45
10 10 2 55.52 5.55
10 10 2 57.86 5.79
10 10 2 57.49 5.75
10 10 2 59.41 5.94
10 10 2 59.97 6
10 10 2 59.69 5.97
10 10 2 59.86 5.99
10 10 2 61.28 6.13
10 10 2 60.67 6.07
10 10 10 27.66 2.77
10 10 10 28.13 2.81
10 10 10 28.77 2.88
10 10 10 28.63 2.86
10 10 10 28.05 2.8
10 10 10 28.02 2.8
10 10 10 28.56 2.86
10 10 10 28.98 2.9
10 10 10 28.61 2.86
10 10 10 29.92 2.99