meteor
meteor copied to clipboard
raystack
Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.
**Is your feature request related to a problem? Please describe.** Our current `Asset` model has below fields to store metadata create & update time. ```go // The timestamp when the...
Build a generic HTTP extractor. A sample config ``` source: name: http config: request: host: http://example.com method: POST path: /v1/employees encoding: json query: foo: bar body: { "fieldA": "valueA", "nestedField":...
**Is your feature request related to a problem? Please describe.** In the current data discovery system, whenever an asset(eg: table/job) gets deleted, the deletion information is not passed down to...
**Describe the bug** When running recipes inside a directory, meteor error messages are not helpful. Even though there is a recipe present, because of recipe errors, meteor returns back no...
**Is your feature request related to a problem? Please describe.** Currently in bigquery extractor, one can exclude datasets/tables based on name. But if the tables/dataset list is huge or if...
```sh # buf # https://github.com/bufbuild/buf/releases/tag/v1.26.1 go install github.com/bufbuild/buf/cmd/[email protected] go install github.com/bufbuild/buf/cmd/[email protected] go install github.com/bufbuild/buf/cmd/[email protected] # meteor # https://github.com/raystack/meteor/releases/tag/v0.9.0 git clone [email protected]:raystack/meteor.git -b v0.9.0 cd meteor && make build go build...
**Is your feature request related to a problem? Please describe.** If an API call fails with a temporary error while running an extractor, we do not retry it. It could...
- [ ] Add unit tests - [ ] Add extractor - [ ] Add README.md in `plugins/extractors/mlflow`, defining output - [ ] Register your extractor `plugins/extractors/populate.go` - [ ]...
**Is your feature request related to a problem? Please describe.** Within the extraction process, there is a concept of batch size but we only ever use the default batch size...