Optimize settings for sustainable write throughput

[saelbrec]

Tests have shown that the currently configured default settings on a vpool do not allow to guarantee a sustainable write throughput. In certain tests it quickly degrades over a time window of 24hrs. The following graph shows on the left half test results with the default values, where as the right half shows test results with optimized volumedriver config values for number_of_scos_in_tlog and non_disposable_scos_factor

At vpool creation time there are 3 factors to take into account.

1. SCOsize can be (4,8,16,32,64,128 MiB)
2. number_of_scos_in_tlog has a default of 4 and is recalculated behind the scenes.
3. non_disposable_scos_factor, decided upon behind the scenes based on the chosen volume writebuffer size. Formula: volume writebuffer = SCOsize * number_of_scos_in_tlog * non_disposable_scos_factor

Constraints we should implement to make sure we configure better balanced values:

• Enforce non_disposable_scos_factor >= 1,5. If smaller advise to either decrease SCOsize or increase volume writebuffer as a value to close to 1 would lead to choking.
• If based on volume writebuffer, SCOsize and number_of_scos_in_tlog(default=4) the non_disposable_scos_factor > 4.0 , we should increase the number_of_scos_in_tlog to maximum the equivalent of 256MiB of data(256MiB/SCOSizeInMiB=number_of_scos_in_tlog) and then use that value to recalculate the non_disposable_scos_factor.

[wimpers]

I'd like to propose a suggestion to the above: use an easy mode and an advanced mode?

• Easy mode: user defines the write buffer size. We derive/calculate some sane values based upon the write buffer size.
• Advanced mode: User needs to enter write buffer size , SCO size, number_of_scos_in_tlog and non_disposable_scos_factor. We enforce whatever is needed like we do when creating policies.

[khenderick]

I kinda like @wimpers his idea. The easy mode asks a few questions and generates a set of works-for-most-setups parameters. The expert mode just exposed (almost) all configuration parameters. This way we can stop changing the default values every month.

[sks126]

@wimpers The following formula in 256 MB is what mean？ "256MiB/SCOSizeInMiB=number_of_scos_in_tlog".

[ngugc]

@saelbrec ，what's the meaning of 256MiB in (256MiB/SCOSizeInMiB=number_of_scos_in_tlog)? Is it just a empiric value? Can I use it in any case? Thanks!

[saelbrec]

The 256MiB in this case is basically the amount of data we address in 1 tlog, how many entries this ends up being depends on the chosen block size. So it is not that I am too concerned with values bigger than 256MiB, it can be used and will work. It's just that other factors like memory consumption, metadata write-out times, write buffer disposability barriers, etc. need to be taken into account. These are no one-fits-all settings, quite some input parameters need to be taken into account, that is why @wimpers suggestion about an easy and advanced mode where one can configure all parameters for his specific use case and aligned with the available hardware is probably the best way forward.

On Wed, Jul 19, 2017 at 3:14 AM, ngugc [email protected] wrote:

@saelbrec https://github.com/saelbrec ，what's the meaning of 256MiB in (256MiB/SCOSizeInMiB=number_of_scos_in_tlog)? Is it just a empiric value? Can I use it in any case? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openvstorage/framework/issues/1687#issuecomment-316243731, or mute the thread https://github.com/notifications/unsubscribe-auth/AMJNxxJ9bNfKGHo7T_7BUUnvJCY9HWJEks5sPViSgaJpZM4ODRer .

[JeffreyDevloo]

Will be picked up after merging the initial changes to add vpool to keep it re-viewable.

[wimpers]

• In the easy mode:
  • max write buffer 20GB
  • In case of easy mode, make sure the number_of_scos_in_tlog is set to 8 (or higher)
  • Enforce non_disposable_scos_factor >= 1,5. If smaller advise to either decrease SCOsize or increase volume writebuffer as a value to close to 1 would lead to choking.
• In case of the advanced mode remove the restriction on the the write buffer (basically anything goes), number of sco's per tlog can be manually entered.

[wimpers]

Decision by PM:

• User can always enter 3 values
  • SCO Size (dropdown) (switch cluster size and sco size from place)
  • Volume Write Buffer (text field, no limitation for size)
  • SCOs per Tlog (default 8, text field which only accepts integer)
• Underneath the non_disposable_scos_factor is shown as information
• In case the non_disposable_scos_factor is lower than 1.5, a warning message is displayed that at least a write buffer of XMB is advised.

The above needs to be changed on the vPool wizard (add, update) and the edit vdisk page.

This also means there will be no easy mode. We always allow users to change all parameters.

[JeffreyDevloo]

We always allow users to change all parameters. except for ... :+1:

[redlicha]

Relevant volumedriver configuration (cf. volumedriver_fs --config-help-markdown):

• volume_manager section
  • number_of_scos_in_tlog: A SCO is non-disposable (dirty) while it's not marked as safely stored on the backend. Marking it as disposable only happens once the associated TLog is safely stored on the backend. Not dynamically reconfigurable.
  • non_disposable_scos_factor: Factor to multiply number_of_scos_in_tlog with which determines the amount of non-disposable (dirty) SCOs permitted per cache namespace (volume). Once that limit is reached, throttling kicks in. Not dynamically reconfigurable.
• volume_router section
  • vrouter_sco_multiplier: Factor to multiply the cluster_size size with to determine the SCO size. This is used as default value for newly created volumes. Not dynamically reconfigurable.

The cluster_size can only be set on vpool creation and must be set to the same value for all members of a vpool - volume_manager/default_cluster_size.

These settings can be overridden per volume with the Python API:

• TLog multiplier:

Docstring:
set_tlog_multiplier( (StorageRouterClient)arg1, (str)volume_id, (object)tlog_multiplier [, (object)req_timeout_secs=None]) -> None :
    Set the number of SCO's in a TLOG.
    @param volume_id: string, volume identifier
    @param tlog_multiplier: None (= volumedriver global value is used) or uint32, the number of SCO's in a TLOG
    @param req_timeout_secs: optional timeout in seconds for this request
    @raises 
          InvalidOperationException (if TLOG size would become invalid)

• non-disposable factor:

In [3]: src.StorageRouterClient.set_sco_cache_max_non_disposable_factor?
Docstring:
set_sco_cache_max_non_disposable_factor( (StorageRouterClient)arg1, (str)volume_id, (object)max_non_disposable_factor [, (object)req_timeout_secs=None]) -> None :
    Set the factor of non-disposable data.
    @param volume_id: string, volume identifier
    @param max_non_disposable_factor: None (= volumedriver global value is used) or float, the factor of non-disposable data
    @param req_timeout_secs: optional timeout in seconds for this request

• SCO multiplier:

[4]: src.StorageRouterClient.set_sco_multiplier?
Docstring:
set_sco_multiplier( (StorageRouterClient)arg1, (str)volume_id, (int)sco_multiplier [, (object)req_timeout_secs=None]) -> None :
    Set the number of clusters in a SCO.
    @param volume_id: string, volume identifier
    @param sco_multiplier: uint32, the number of clusters in a SCO
    @param req_timeout_secs: optional timeout in seconds for this request
    @raises 
          InvalidOperationException (if SCO size would become invalid)

[wimpers]

@JeffreyDevloo didn't we implement this already. I believe I saw screenshots of this.

[JeffreyDevloo]

No. The front-end side has the option to configure them, but that is currently disabled. The API side of things has not be done

[wimpers]

Can you put an estimate on the API side?

[sjovdnbos]

GUI changes:

• [x] add + reconfigure vpool: #2181

API changes:

• [ ] reconfigure api call