hash icon indicating copy to clipboard operation
hash copied to clipboard

H-2753: Update Renovate scheduled for LLM provider SDKs

Open vilkinsons opened this issue 1 year ago • 3 comments

🌟 What is the purpose of this PR?

We want to ensure we're always using the latest version of the LLM provider SDKs we rely on, without:

  • waiting the normal 3 day period npm unpublishing safeguards would normally have us delay updating for
  • having to manually trigger these to be updated via the repository's dependency update dashboard.

Further rationale in this chat thread (internal link). To this end, we want PRs to be opened, but not set to automerge, so that their changelogs can be reviewed by a member of our team (and any relevant features taken advantage of) before merging in.

As well as supporting @openai/ and @anthropic/ npm namespaces, this PR pre-emptively adds handling for the groq-sdk package to this Renovate schedule, as well.

🚫 Blocked by

A review of the Renovate logic used here, to make sure it functions as expected, would be appreciated. In particular my setting of minimumReleaseDate to "0 days". If there is a better solution for overriding the default 3-day wait we apply to npm updates, please feel free to suggest a change.

vilkinsons avatar May 22 '24 10:05 vilkinsons

@TimDiekmann Could you make this change and re-approve? Thanks!

vilkinsons avatar May 22 '24 10:05 vilkinsons

@TimDiekmann Could you make this change and re-approve? Thanks!

Done

TimDiekmann avatar May 22 '24 11:05 TimDiekmann

Benchmark results

@rust/graph-benches – Integrations

scaling_read_entity_complete_one_depth

Function Value Mean Flame graphs
entity_by_id 50 entities $$1.56 \mathrm{s} \pm 4.86 \mathrm{ms}\left({\color{red}463 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$32.4 \mathrm{ms} \pm 170 \mathrm{μs}\left({\color{lightgreen}-38.111 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1 entities $$21.6 \mathrm{ms} \pm 148 \mathrm{μs}\left({\color{gray}0.746 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 25 entities $$76.4 \mathrm{ms} \pm 513 \mathrm{μs}\left({\color{gray}3.14 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 5 entities $$25.9 \mathrm{ms} \pm 325 \mathrm{μs}\left({\color{gray}0.746 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity

Function Value Mean Flame graphs
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1 $$16.9 \mathrm{ms} \pm 211 \mathrm{μs}\left({\color{gray}-2.966 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1 $$17.0 \mathrm{ms} \pm 189 \mathrm{μs}\left({\color{gray}-2.810 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2 $$16.7 \mathrm{ms} \pm 155 \mathrm{μs}\left({\color{gray}2.13 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1 $$17.2 \mathrm{ms} \pm 204 \mathrm{μs}\left({\color{gray}4.09 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1 $$16.1 \mathrm{ms} \pm 156 \mathrm{μs}\left({\color{lightgreen}-6.918 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1 $$17.4 \mathrm{ms} \pm 231 \mathrm{μs}\left({\color{gray}1.43 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1 $$16.7 \mathrm{ms} \pm 195 \mathrm{μs}\left({\color{gray}-1.475 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1 $$16.8 \mathrm{ms} \pm 243 \mathrm{μs}\left({\color{lightgreen}-29.243 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1 $$17.2 \mathrm{ms} \pm 218 \mathrm{μs}\left({\color{red}7.05 \mathrm{\%}}\right) $$ Flame Graph

representative_read_multiple_entities

Function Value Mean Flame graphs
link_by_source_by_property depths: DT=255, PT=255, ET=255, E=255 $$2.01 \mathrm{s} \pm 9.07 \mathrm{ms}\left({\color{gray}-0.454 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=2, PT=2, ET=2, E=2 $$1.07 \mathrm{s} \pm 4.58 \mathrm{ms}\left({\color{gray}0.030 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=2, ET=2, E=2 $$1.05 \mathrm{s} \pm 7.39 \mathrm{ms}\left({\color{gray}-0.710 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=0, E=2 $$98.3 \mathrm{ms} \pm 496 \mathrm{μs}\left({\color{gray}0.224 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=2, E=2 $$429 \mathrm{ms} \pm 1.57 \mathrm{ms}\left({\color{gray}0.508 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=0, E=0 $$61.5 \mathrm{ms} \pm 314 \mathrm{μs}\left({\color{gray}-0.347 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=255, PT=255, ET=255, E=255 $$2.90 \mathrm{s} \pm 11.2 \mathrm{ms}\left({\color{gray}-0.273 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=2, PT=2, ET=2, E=2 $$995 \mathrm{ms} \pm 6.62 \mathrm{ms}\left({\color{gray}0.021 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=2, ET=2, E=2 $$982 \mathrm{ms} \pm 3.70 \mathrm{ms}\left({\color{gray}-1.233 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=0, E=2 $$41.2 \mathrm{ms} \pm 197 \mathrm{μs}\left({\color{gray}-1.455 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=2, E=2 $$363 \mathrm{ms} \pm 1.86 \mathrm{ms}\left({\color{gray}-0.686 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=0, E=0 $$37.1 \mathrm{ms} \pm 139 \mathrm{μs}\left({\color{gray}-1.394 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity_type

Function Value Mean Flame graphs
get_entity_type_by_id Account ID: d4e16033-c281-4cde-aa35-9085bf2e7579 $$1.41 \mathrm{ms} \pm 4.97 \mathrm{μs}\left({\color{gray}-0.179 \mathrm{\%}}\right) $$ Flame Graph

scaling_read_entity_linkless

Function Value Mean Flame graphs
entity_by_id 1000 entities $$3.41 \mathrm{ms} \pm 30.3 \mathrm{μs}\left({\color{gray}2.09 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 100 entities $$2.61 \mathrm{ms} \pm 14.6 \mathrm{μs}\left({\color{gray}3.22 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$2.46 \mathrm{ms} \pm 11.4 \mathrm{μs}\left({\color{gray}-0.535 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10000 entities $$13.9 \mathrm{ms} \pm 134 \mathrm{μs}\left({\color{gray}2.37 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1 entities $$2.47 \mathrm{ms} \pm 12.2 \mathrm{μs}\left({\color{gray}0.452 \mathrm{\%}}\right) $$ Flame Graph

scaling_read_entity_complete_zero_depth

Function Value Mean Flame graphs
entity_by_id 50 entities $$4.50 \mathrm{ms} \pm 21.2 \mathrm{μs}\left({\color{gray}0.837 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$2.72 \mathrm{ms} \pm 16.9 \mathrm{μs}\left({\color{gray}0.116 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1 entities $$2.51 \mathrm{ms} \pm 16.8 \mathrm{μs}\left({\color{gray}-0.443 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 25 entities $$3.17 \mathrm{ms} \pm 20.7 \mathrm{μs}\left({\color{gray}1.22 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 5 entities $$2.52 \mathrm{ms} \pm 11.4 \mathrm{μs}\left({\color{gray}-0.205 \mathrm{\%}}\right) $$ Flame Graph

github-actions[bot] avatar May 22 '24 11:05 github-actions[bot]